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ABSTRACT * • • - 

Thirty techniques . and tools ■ for validation^ verification, and testing 
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effectiveness and usability, applicability, an estimate of the learning time 
and training, an estimate of needed resources , and references. 
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v 1.-1 Introduction / - 

The Institute for Computer Sciences and Technology (ICST) carries out the 
^£eiiot5ng .responsibilities under P.L. 89-306 (Brooks Act; to improve the 
Federal Government's management and use- of ADP: 

. ^ j ' ' • 

' 5 "develops 'Federal automatic data processing standards; 

. o provides agencies' with, scientific and technological advisory services 
relating to ADP; * 

o undertakes necessary research in computer sciences and technology. ' 

In partial fulfillment of Brooks Act responsibilities^ ICST issues Special 
Publications (S.P.). This document is a reference guide for techniques and 
tools which may be used in conjunction with a validation, verification, and 
testing (V,V&T) methology. . 

The document consists of (three sections: 

6 A suggested methodology for the selection, of V,V&T techniques and tools. 

\ o Summary matrices by development phase usage, a table of techniques and tools 
with t associated keywords, and an alphabetized table of keywords with 
associated techniques and tools. 

6 Description of 30 V,V&T techniques and tools. 

This document can be used independently as a reference or can be used in 
conjunction with "Guidelines -on Planning for Software Validation, 
Verification, and Testing" (to be published as a FIPS PUB in 1982). 

A glossary, ^eluded as Appendix A, defines- terminology used in this document. 

2.1 A Suggested Methodology for the Selection of V,V&T Techniques and Tools 

The FIPS PUB ^Guidelines on Planning for Software Validation, Verification, 
and Testing" (to be -published) explains - the role of V,V&T in software 
development, stressing an integrated approach. V,V&T planning by identifying 
goals, -determining factors which influence the V,V&T activity K selecting V,V&T 
"techniques and tools, and developing a detailed V,V&T plan are explained in 
detail. This "document is particularity helpful' in the selection of techniques 
and tools. 

Selecting techniques and tools begins with the determination of a goal - a 
[specific, measurable outcome. For example, ^90 percent statement execution is 
a goal. Once a goal is determine, the selection matrices (sefction 3$ are 
utilized to see if a technique or tool is applicable to the selected, goal. 
For the example above, statement coverage Us checked during code expiation. 
Referencing the code selection matrix, one finds statement coverage^ Next, 
"the alphabetized keyword table (section 3)*is searched for the appropriate' 
keyword(s)-. \For -the exanple, the* tool for statement coverage is found to be 



^st coverage analyzers. The last step is to reference the technique and tool 
S25S8KL* C ! ,0 J^!f" 4 > confirm; , that the technique or .tool does 

accomplish the desired goal. For, the example under test coverage ' analyzers, 
toe statement ^Completenesses measured: in terms of the branches, statements 
or other elementary constructs which are ^ used: during- the" execution of the 

.program over the tests", conf iros that a statement coverage analyzer measures 
the completeness of statement execution.-^ . ^ assures 

2.2 Selection Aids : . ' 

Tables 3.1-1,3.1^2, and 3. 1-3 -separate techniques and tools into the broadly 
defined software development. phases: requirements, design, and code. 
< : f :■ • . 
The purpose of a selection matrix is to suggest possible techniques or tools 
;2h * *«?°v« a d ? v J l0 P nent Phase. /The goal is stated', (directly or 
. indirectly) in terms of the form or. content of a development product 
^SSFS 0 *?' df 31 ? 11 .. code).1 The matrices list V,V&T techniques and tools 
applicable to analyzing the -form or content of a product. Specifically 
manual and; automated, static' analysis techniques and tools aid in analyzing the 
fora-of ;each of the three products. Dynamic and formal techniques and tools 
aid m analyzing the- semantic content of each. of the products. 

table 3.1-4 lists, alphabetically, the keywords and the associated technique 
or tool. It may be used to identify characteristics of the technique or tool 
from one of the three matrices in Tables 3.1-1, 3.1-2 or 3.1-3. 

Table 3.1-5 lists each technique or tool described in section 4 with 
applicable keywords. It may also be used to identify. the characteristics of a 
"technique or tool. 

The reader with sufficient knowledge may skip Tables 3.1-1 through 3.1-5 and 
go directly to the technique and tools section. 

3.1 Selected Matrices and Keyword Tables • »' 

The pages that follow contain three selection matrices: : 

.Table 3.1-1 - Requirement Specifications * 
Table 3.1-2 - Design Specifications - 
Table 3.1-3 - Code ' k 

and 

Table 3.1-4 - V,V&T Techniques and Tool Keywords 

-Table 3.1-5 - V,V&T Techniques and Tool with Keywords ^ 



ANALYSIS TYPE . AUTOMATED TOOLS 



MANUAL" TECHNIQUES 



REVIEWS 



V 



tatic 



Requirements 
tracing aids 
(Note 1) • . 
Cross-reference 
Data flow analyzer 



Dynamic 



Formal 



NOTES 
1) 



Requirements 

analysis 
Cause-effect 

graphing 
Assertion generation 
Data flew analyzer 



^Assertion generation 



Requirements 
tracing aids 

(Notes 1&2) 
Inspections 
: r .Selected manual 

application of 

techniques listed 

in column one 1 
(Note 3) 

Assertion generation 
* (Note 4) 
Specification-based 

functional .testing 

(Note 5) " 
Cause-effect graphing 

(Note 5) 
Walkthroughs - 

Formal verif ica tion 
(Note 6) 



Inspections 
Peer review ' 
Formal reviews 



.Walkthroughs 
Formal reviews 



The requirements indexing and cross-jreferencing schemes are established and 
documented as part of the- requirements specification. . J ' ( 
2) Requirements tracing may be performed through a totally manual process. 
3> Certain techniques may be manual^ applied to 'small applications or on 
^ selects portions of a given specification. This requires planning .and 
and preparation. The larger the amount of information being analyzed, 
the greater the probability of error. ■ " 

Assertion^heratioh is performed either for later analysis using an 
assertion processing tool, or for manual analysis as an adjunct to testing. 
This is a test data generation technique/ tool. - 
Axiomatic specification is necessary to support analysis. H 



4) 

5) 
6) 



TABLE 3.1-1 • 
SELECTION MATRIX I REQUIREMENT SPECIFICATION 



ANALYSIS TYPE*-, AUTOMATED TOOLS 



MANUAL TECHNIQUES 

" i» , 



RBCEEWS 



Static 



Dynamic 



Formal-, 



NOTES 
1) 



Requirements 
tracing aids ^ 
Cross-reference ^ 
Data frow analyzer 



Cause-effect 
graphing 



^Analytic modeling of 

software designs 
o (Note 6) * 
"Global roundoff 

analysis' of 

•algebraic processes 
N (Note 5)" 
Formal verification 

(Note 8) * 



Requirements' • * S . 
tracing (Note 1) 
Inspections 

- Selected manual j 
application o£; > : 
techniques listed iij ;. 
colunn ong 

(Note 2) ^ r 

Assertion generation -"v 

(Note 3) 
Specification-based 

functional testing 

(Note .4) '. ' 
Cause-effect graphing 

(Note 4) 
Walkthroughs 

Algorithm analysis * 
Formal verification . 
(Notes 7&8) 



Inspections 
Peer review • 
' Formal reviews 



Walkthroughs 
•Formal reviews 




2) 



Requirements tracing may be performed through a totally manual process. 
Certain techniques may be manually applied to small applications or on 
; selected-portions* of a given specif icatidh. This requi»es planning and 
preparation, .the larger the amount of information being analyzed.- " 
the greater the probability of 'error. • 
3) Assertion generation is performed either for later analysis using* an' 

assertion processing tool, or for manual analysis as an adjunct to testing 
This is a test data generation technique/ tool. ■ ' • r 

Analyzes an algebraic algorithm,- independent of a given* level of" 
specification and; therefore is applicable to a design or code level ' • ••' : - 



4) 
5) 



specification. 

6) Requires the mant 

7) Axicmatie specif i 

8) Formal verificat 
have been develoi 



eyej 

-development of, a model, which is then'ru'n. 
-Jtion*is^ne^ssary,:to support analysis. 

a primarily manual exerciserthough supporting tools 
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SELECTION" MATRIX II s DESIGN SPECIFICATIONS 
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ANALYSIS TYPE AUTOMATED TOOLS . 



MANUAL TECHNIQUES 



REVIEWS 7 



Static 



Dynamic 



Formal 



Requiranents * • V 
tracing..* ,/\v^;$-:'::^ 

Cross-reference ' > "« 
Data f low ^eOT3^r7 

C<»trQl s^ctuce 
-analyzer ^ 

Interface checkers 

Physical units- 

-checking 

Code auditor T 

(kxnparator c 

Test data generator 



Assertion processing ; Assertion generation 
Test data generators * (Note 3) 



vlRequirenients ■ v Inspections 

- tracing aids, (Note j) Peer review r / 
Infec tions ' ; Formal reviews 

r Selecteff^ nianuai ~~ - ■ 

application of ■ r' 
techniques listed in* 
coluna one ■ ' ; 
(Note 2) ' " * 



Walkthroughs 
Formal reviews 



Test support 

facilities 
Test coverage 
■ '. ahalx^is. ; ';■•>'* ■ V 
Mutation analysis < 

CNote 4)V c * •« 

Interactive test aids 
Execution time 

estimator/anal3^er(Note 5) 
Software monitorCNote 5) 
Statement coverage ; 
Symbolic evisGLuatjpn 



Regression testing ' 
.(Note jS);^ ' V; 
WaikttiT^ 



Formal verification > Formal verification 
(Note 7) - ■ ■ (Note 7) ' 



notes ; y-'~ • :■.<■-■ • 

1 ) Requirements tracing may be performed through a totally manual process. 
Certain techniques may be manually applied to small- applications or on ; 
selected portions of a ^ven sj^if ication. Ihis r^uira planning and 
ancj preparation. . The larger the amount of information being analyzed, 
the greater the probability of error.' • ^l^TT"- 1 " t 

Assertion generation is performed either for later analysis using an 
assertion processl^^ t^ or for manual an^y^s. a an adjunct to testing. 
The objective /o^nutatidn an^ysis is to help, assess the sufficiency of the 
test diata. : ' '^Y'- yy' yyYY:' ' y ■[ :*V\.-:r .v\". : : : -- ;VV ": ' 7 ' •• ''•V ' ' V\- 
A^ist in testing the satisfaction of performance related requirements. 
Testing af iwxlif icat^n of tested software, i.e.'f retesti^; r.~ - ^ . 
Fotinal yerif icati^ primarily manual exercise though supporting tools 
have been developed. - - /■ 



2) 



:3J 

M 

5) 
6) 

-7) 



V ^ > • TABLE 3.1-3' - 

SEij:CTioN;;M^RiX'in code 
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accuracy- analysis " 
algorithm efficiency., . 3' 
amount of space (memory, disk; etc.) used 
assertion ^ (Cra 0 ^ tion s>' <tone , 
bottlenecks* ,.' V 

'*'.', * • - 

boundary- test cases 

" branch and path identification . 

branch testing . • ; . 
'«-• call graph . " .'*• •"• 'Jr. ■ ■: : ' ■ 

^check list •" ^ .. 

. ; code' reading :•; ' 

ccmple.teness , of . test data 
computational upper bound, how fast 
consistency- in computations 

- correspondence between actual and formal 

- parameters - , 
data (characteristics 

•dynamic testing of "assertions 
** environment simulation 

eyaluation along program paths 

execution monitoring 

execution sampling 

execution support 

expected -inputs, outputs, and 
intermediate results ; ■ ."'.v 
• expected, versus actual results 
. file (or other event) sequence errors 

formal specifications ■• ... * 

functional interrelationships 

global information flow 

go/no go decisions 

' hierarchical interrelationships of modules- 
information flow consistency ,'„'.« 
inspections ....... 

' inter-module' structure 
loop invariants ' ' 

manual simulation 

module invocation 

numerical stability 



techniQUfr/TVyfl .... 

algorithm analysis ; 
algor ithin^analysls^ " 
algorithm analysis- 
algor ithm analysis 



~~ Tiassertion processing 
V analytic deling of 
-' ; software designs \ 
specification-based functional 
testing . 
control stnicture an£lyzer« 
test coverage' analyzers 

control striacture analyzer 
inspeptiohs : • 
peer review * ' 

mutation ''analysis 
algorithm analysis 
physical units testing 
interface checker >\ 

assertion generation 
assertion processing 
test support facilities 
symbolic^ execution 
software ^ monitors . * r 
software monitors 
test support facilities 
- assertion generation 

comparator : 
data flow analyzer 
assertion generation 
requirements : analyzer 
interface checker . 
formal previews 
• control structure, analyzer 
requirements analyzer .'-J. 
peer review 

cross-reference generate 
assertion generation . 
walkthroughs 

control stracture analyzer 
global roundoff ".analysis of • 
algebraic processes 



: TABLE 3.1-4 : 'J :"•< 
V, V4T IfiGHHpE AND TOOL KEIWORDS 



ERIC 



Keywords 

path testing 
performance analysis 
physical units , 
portability analyzer 
i>rogram-^^ution-characteristics^- 



prpof of correctness : ' ~ / 
regression testing ' * 
requiremente indexing ~ - . ., ' 
requirements specification analysis 
requirements to design correlation 
requirements walkthrough 
retestdLng after changes / 
rouncJrrobin reviews 
rounding error propagation 

selective program execution 
standards -checker 
statement coverage . . 

Ptement testing \, . 

tus reviews 
tern performance prediction 

technical review 

test, caise preparation (definition am 
specification) \ 
test data generation 



test harness^ -T "* - - " C 
testing thoroughness - 
type checking 

uninitialized- variables 4 °". ,'• 
unused yafi^>lfes ' 
variable .references 
vartablei ^ 

verification- qf algebraic confutation 
walkthroughs " 




Technique/Tool *. 

test coverage analyzers 
requirements analyzer 
assertion generation 
code auditor - 

execution - time~estimator/^v 
analyzer ^ 
software monitors y : 
formal verification \ 
comparator 
requirements tracing 
. cause-effect graphing 
requirements tracing * 
requirements analyzer 
regression testing 
peer reviews 

global routidof f analysis of 

algebr&ic processes 
interactive test aids - 
code, audi tor - 
test coverage analyzers 
test coverage analyzers 
fortnal reviews 
analytic modeling of 

software designs 
peer review . 
test data generators 



mutation analysis 
specification-based functional 
testing r . V. 

test support facilities ' v ... 
test coverage analyzers. ; - : 
interface checker •* 
data flow analyzer . *V 
data-flow analyzer . .. 
cross-reference generators 
interactive test aids ■ 
symbolic execution 
peer reviews / > 
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TactoimWf nrrt 

Algorithm Analysis 



Analytic Modeling of 
Software Designs 

Assertion Generation 



Assertion Processing 
Cause-Effect Graphing 
Code Auditor 

* 

Comparator 

Control Structure Analyzer 



Cross-Ref erence Generators 
Data Flow Analyzer 



XssassEte. 

algorithm efficiency 
amount of work (CPU operations) done 
computational upper bound, how fast 
amount of space (memory , disk, etc .) used 
accuracy analysis 

system performance prediction 
bottlenecks 

formal specifications 

data characteristics * 

physical units 

loop invariants 

expected inputs, 

outputs and intermediate results 
assertion Eolations 

dynamic ' testis of assertions , 

test case design, using formal specification 
requirements, specification analysis 

* • 
standards checker 
portability analyzer 

regression testing 
'expected versus actual results 

call graph." - ^ 

hierarchical interrelationships of ' 
modules ' 
module invocation .. ■ '" 
^ branch and path identification 

inter-module structure 
variable references 

uninitialized variables v ^ 
unused variables - 

file (or other event) sequence errors 



Execution Time Estimator/Analyzer program execution characteristics 



Formal Reviews 



Formal Verification 



go/no go decisions 
status reviews 

proof of correctness 



TABLE 3.1-5 
V,V&T TECHNIQUE/TOOt, WITH KEYWORDS 



ERIC 



16 



Page 9 



Global Roundoff Analysis of 
Algebraic Processes 

Inspections 

Interactive Test Aids 

Interface Checker 



Mutation Analysis 



Peer Review 



Physical Units Testing 
.Regression Testing 

Requirements Analyzer 

". ... t 

Requirements Tracing 
Software Monitors 

Specification-based Functional 
Testing 

Symbolic Execution 
Test Support Facilities 



\ nunerical stability v . : - - 

. minding error propagation 

: ^eck list , ;'v 

s^Lective program execution 
variable snapshots/tracing 

correspondence between actual and formal 
parameters 

type checking * . 
global -information' flew ^ 

test data^ generation 
completeness of test d^ta 

technical review 
code reading \ 
round-robin reviews i 
walktiiroughs V 
inspections , V 

consistency in computations 

retesting after changes 

functional interrelationships 
information flow consistency 
performance analysis 
. requirements walkthrough 

requirements;; indexing : 
. requirements to design correlation 

execution, sampling % 
execution monitoring \ 
r program execution characteristics 




test data generation 
boundaCT-teist* cases 



evaluation along program paths 

verification of algebraic computation 

- ■ * ' ■ ■ \ * 

test harness 
execution support 
environment simulation 



TABLE 3.1-5 (Continued) N 
; V f V&T TECHNIQUE/TOOL WITH KEYWORDS 
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TCChnlQW/TPPl Keyword^ / 

Test Coverage Analyzers . ^ branch testing ~ 

statement testing 
statement coverage 
path- test ing 



testing thouroughness 

Test Data Generators test case preparation (definition 

w »• and specification) 

Walkthroughs \ manual simulation 

TABLE 3.1-5 (Continued) • 
V,V&T TECHNIQUE/TOOL WITH KEYWORDS ' 
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4*1 INTRODUCTION TO tECHNIQUE AND TOOL DESCRIPTIONS - ■ 

' , ' : - V • . V P ' ' 

Each technique and topi description is alphabetically presented in a -standard, 
format. The following table describes the entries for each where n n n is the m * 
section 'number. . * * -■■*'■* ° ~ • 

~^47n.V. ; Name " ~ ' : ■ r T~ ~ TT ^- ~ : ~~ 1 ' . 

Ibis is the accepted title, or when an appropriate one does not exist, an 
-invented title. 

4.n.2. Basic Features / 
A short description of the technique or tool. 

4.n.3* Information Input '■ , ~ • 

A description of the input required for use. 

4.n.4. Information Output ' ■ * -.\\ . * ' 

A description of the results o£ the' technique or the output of the tool. 

4.n.5. Outline of Method 

A.brief list of the actiofls that a user is- expected to perform. 

4.n.6. Example ' » 

An example to illustrate the inputs, outfits, and the method. 

4.n.7. Effectiveness^ . y 
A brief assessment -of the effectiveness and usability.,- including underlying' 
assumptions and difficulties that can be expected in practice. * v- 

m 4.n.8. Applicability . . V ■ \ " 

' An indication of the situation* in which the technique is likely to be useful. 

— 4.n.9. Learning——" ■— ,- 7 ^ % — i- . . ' . / ' , 

An estimate of the learning time and training needed to use the technique or 
to61 successfully* V. . ■■\-\ 

4.n.10. Cost, 

An estimate of the resources needed.- 

4.n.11. References f 
Sources of additional information. 

; . • ., < * ■ ' v " 

TABLE 4.1-1 . / .. * . 

TECHNIQUE AND TOOL DESCRIPTION ENTRIES 



ERIC 



'• ,'• ' *• Page 12, 

4.2.T. Name. ALgpritim Analysis. : - " 

i; 2 :^ features, ..iwo phases of. "algorithm 1 analysis can be 

distoguished: . "a; .priori: analysis" ana. "a posteriori testing." In a .priori 
^^J/ U0CttiGa ^ 0 L Sa ^ relevant parameters) is devised" Sch'Su^dl Se 
'SS^f'? USG ■■°L> 1 S? and* spac e to compute an accept able soluti on.' iy 
analysis assumes a model of computation such as: . a Turing macfaineVJUM 
SSKS'-* 00688 -^f 0 ^: general purpose machine, etc Two general kinds^of 
E? ^2i^ U ^ Uy tr f atedS ■, (T) *'«' Particular algorithmT and 
i?!kXf y iS of „ a .S aS3 of algorithms. In a posteriori testing actual • 
wSS S C ?s^ut^^ the algorithm's consumption of time. and. spaS 

4.2.3. Information input. .... , • 

. a. Specification of algorithm . 
P. Program representing the algorithm 
. 4.2 Jr. Information output. , _ 

a. A priori analysis ' 

Confidence of algorithms 1 validity. 

Upper and lower computational bounds \ ; ^ 

Prediction of space usage 
Assessment of optimality 

b. A posteriori testing \ ■ : 
. Performance profile 

4.2.5. Outline of method. - 
a. A priori analysis 

i^orithms^are analyzed with the intention of improving them, if possible, and 
hqt bfSsS* ^ D0Dg several available for a problem. The following criteria > 

Correctness/ 
Amount of work done 

Amount of space used . • - 

Simplicity - - r . 

Optimality 

Accuracy analysis J , ; ., . . w 

Correctness . There are three major steps* involved ^ - 'establishing the 
correctness of an algorithm. . • • %: ; : 

CI) Understand that an algorithm is correct. if, when given a valid input, 
it computes for a finite amount of tlroe and produces the right answer. 



2o 




• ■ . ,. ; fige 13 u 

(2) Verify , that the^athenatical properties of the me€hod and/or formulas-* 
used toy the algorithm are correct. \ r - , 

(3) Verify mathematical argun^t that the instructions* of the algorithm 

do produce the right answer and do terminate* ' : - :\ - ' > 

AnsyQ£ i2£ i£2Ek \dSD£* A priori /an^ysis ignores all of the factprs which 
are > machine or programing lanjguage dependent and concentrates on 
determining the order of .--'magnitude- . of '-the, frequency of execution of 
statements.. For denoting the upper bound on an algorithm, the (^notation 
is used. The following notational symbols are used in 'the following 
description: / "exponentiation ; ; [^subscription./ - 



Definition, f(n) - Q(g(n)) if and only if there exist 
constants C and n[o] such Jftat f(nKC g(n) for all r£p[o]. 



two positive 



The most common' *• computing times for algorithms, are: 0(1)<0(log 
n)<0(n)<0(nlqg n)<0(n«2)f^0(n«3) and^2»»n), 0(1) means that the nunber 
of executions of basic operations *Isn.xed and hence the total time is 
bouhded^by a constant. The first six orders of magnitude are bounded by a 
polynomial. However, there is no integer such that n**m bounds 2**n. Ah 
algorithm whose computing time; has this property . is said 'to require 
exponential time; There are notations for lower bounds and asynptotic 
bounds - (see reference (4) for . details). The *tera "complexity" is the 
formal term for the amount of work done, measured by some complexity (or 
cost) measure* w '"*r . ; * " 

In gdaeral the amount of work done by an algorithm depends on the size of 



input. In seme 
t»rticular input. 



cases, the nunber of 
Seme examples of size are: 



Problem ~ : 
1i Find X in a list of names \ 

2. Multiply two matrices 

3* Solve a system of linear equations 




■ations may depend on the 



Size:6f Input 
founber of names in the 



The dimensions of the 
Prices . 
number of equations and 
solution vectors - * 7^. 



to handle the-sil^tibh of- to^ affecting- the performance *cf'i an 

algorttto, two approaches ^ are used. The 

average approafeh assunes a distribution of inputs : and*then calculates the 
nunber of operations ;performed for each type of input in the distribution 
and then computes, a weighted- average. Ihe worsfc^se approach calculates 
the neximun nunber & basic operations performed on any input of a fixed 
size* . •• /' >:■■] 7 - * ,>.v 

imoBBt jfif ^SnaSfe ilasd. The number of memory c£Lls used by a program,^ like 
the; nunber of seconds required to execute a program depends on the 
particular -inplementatipn. However, seme conclusions about space usage 
- can be made by examining the algbrithm. A program vill require storage 
space for theinstractio^ and variables used by~ the 
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program, and the ♦input data. It . may also use seme work space for 
- .manipulating the data and storing information needed to carry out its 
-^computations. Hie.' input: data itself may be * cepr esentable in several 
t: 'forms, seme which require more space than others. If the input data has 
... V one ; natural .form - for example, an array of numbers or a matrix - then we 
analyze the extra space used aside* from the program and the- input T Tf the* 
amount of extra space is v constant with respect to the -input size, the 
algorithm is said to work "in place". 

Simplicity. It is often, though not always, the case that the simplest 
and most . straightforward .way of solving ' a problem is not the most 
efficient; -Yet simplicity in .an algorithm is a desirable feature. It may. 
make verifying the correctness, of the algorithm easier, and it makes 
writing, debugging and modifying a program for the algorithm easier. The 
time needed to produce a debugged program should be considered when 
choosing ^an algorithm, : but, if the program is to be used very often, its - 
efficiency will probably be the determining factor in the choice. 

Opt-iTna^itv- Two tasks must be carried out to determine how much work is 
necessary and sufficient to solve a problem. 

C.1) Devise, what seems to be an efficient algorithm; call it* A. Analyze A 
and find a function such that, for inputs of size n, A doestat most g(n) 
basic operations. V 

(2) For some function, f, .prove a theorem that for any algorithm in * the 
• class under consideration there " is some input of "size h for which the 
; r , algorithm inust perform at least f(n) basic operations.' • - .'. 

If the functions g and f are equal, thto the algorithm A is optimal- 

Accuracy analysis. The computational "stability of* .an algorithm is 
verified by '. determining that the integrity of round off accuracy is 
maintained. - It is done manually at the requirements ' or specification 
level/. ^ : :>::"'•- ' '■■J-;. '•'•';•. '. • : ■:/-■: ■■. 

' b. A Posteriori Testing " ' " ^ ■■■ " % v .. ' 

Once an algorithm has been analyzed," the next step is usually the confirmation 
of the analysis. The confirmation process consists first of devising a 
program for the algorithm on a particular computer. ) After •'. the program is 
operational, the next ' step is producing a "performance profile" ; that is, 
determining the precise amounts of time and storage the. program will consume. 
To determine time consumption,/ the. ccmputer clock Is used. Several data sets 
of varying size are. executed .and a performance profile is- developed and 
compared with the. predicted curve. • •':.-.';-" •.*-*- s v * 

A second way to use the .computer's timing capability is to take two programs 
which "perform the same task' whose -^orders- .-of magnitude are identical and 
compare them.as they» process -daita. The show which, if 

either, ; program is faster. Changes to a program which do not alter the order 
of magnitude but which purport to speed up the program also can be tested in 



' this way. ... 



!^'S ' &a ?£ le - .^CKSORf is a recwsive sorting' algorithm (5). " Roughly 

SSf' ^ r S r ^ a? f? S T, the 3 ^ d splits the me'into ^'subsebtionsror 
subfiles, such that all keys in the first section are smaller than all keys in 
the second, section. Then QUICKSORT, sorts the'two subfSs recursively '(i e 



Tby the same method), with the result that the entire file is sorted. 

Let A be^tfae array of .keys and let m and n be the indices of the first and 
SLT^Eri respectively, in *• subfile which QUICKSORT: is currently 
fS^e ^JSffi^iV 1 ^ n = k ' The PARTITION algorithm : chooses a key K 
••2?.J5! f??fS e 1??, rea !: ranges ^ entries, finding an integer j such Sat 
for n*i<j, Mi)& A(j) = K; . and for j<i^n, A(i)^C. k iT then in Its 
correct position and- is ignored in the subsequent sorting. 

QUICKSORT can be described by the foUcwing .recurkve algprithm: • ' • " 



QUICKSORT CA,m,n) 
if. nKn then do 



PARTITION (A,m,n,i,j) 

QUICKSORT (A,m,j) 
. QUICKSORT (A,i,n) 
SDH 



• Figure 4.2.6-1 QUICKSORT . 

The PARTITION routine may choose as K any key in the file between A(m)' and 
. f ^? rv sa?I ^ >:Lio J 1: y* " let ' K A(m) V An- efficient partitionSg^a^orithm uses 
-twowinters, i and j, initialized to m and n+t, respktive^ and^SSns^b? 

entry, ine location AC i) is. filled by decrementing n until A( i )<v and '*" 
copying ACj) into . ACi). Now AQ) is filled bylicrSing^^tif l(iS 
f d ,^ n i co ^g m into" A(j) . JMs procedure coh&uSu^L SvaTues^f 
™ d j "f^ then K is^pft in toe last' place. Observe that PARTITION 
compares: each key except toe^original in M to K, so it does 
comparisons. See (5) for further- details. } I ' * 



n-m 



tofit £^ inalyjsis.lf when PARTITION- is executed A(m) is the largest kev in 
S>^^^ b ^ % t if? A(m)iA(i) for^m^n), then PARTITION will mov^ 
it to the bpttcm to position A(n> and partition,the file into one section with 
(al1 " ^ uS e *>ttcm one) and -one section with no ^entries. -All. 
thaV has been .accomplished is . moving the " maximum entry to the bottom 
Sifflilarly, if the smallest entry in.the f Ue Is in^osition A(m), PARTITION 
will simply separate it from the r^est of the list; leaving ri-m items still to 
Be sorted.- ^Tnus if . the input r is arranged so that each -time PARTITION is" 
SS^' i (m) - lar8e f fc (or ^ snallest), entry -in the secuS^being 
£?2kJ?5 Jet P.= the number of keys rf in the unsorted section, then 

the number of comparisons done is 
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Average BetoiflC A^lisis. if a • sorting algorithm removes at most one 
Aversion f ran toe permutation of the keys after each comparison, then it must 
do at leastJn«2-n)/4 comparisons on the average. QUICKSORT, however, does 
not have this restriction., The PARTITION algorithm can move keys across a 

£S2 e S J5S^x2E'?^? ntire me » eliminating up to n-2 inversions at one 
time... .QUICKSORT deserves its. name because of- its average behavior — — — 



Consider srsituation in which QUICKSORT works quite well. Suppose that each 
time . PARTITION is executed,. < it splits the file into two roughly equal 
subfiles.- To simplify the computation, assume that n = 2**p -1 for seme p 
The number of comparisons done ,by QUICKSORT on a file with n entries under 
tftese assumptions- is described by the recurrence relation . 



R(p) = (2*«p)-2+2R(p-1) 
R(1) = 0 



^ SiSwSS ^ . r <pV (2»^)-2, are n-1, the number of comparisons done 

by PARTITION the first time. The second term is the number of comparisons 
done^ by QUICKSORT to. sort, the two subfiles, each of which has (n-1)V2, • or 
(2**(p-1 •)•). -1 , entries. Expand the recurrence relation to -get ' 



thus 



R(p) = (2*«p)-2+2R(fHl) = ;(2**p)-2+2(2**(p-1 )-2)+4R(p-2) 
.= (2««p)-2+(2»»p)-4i.(2»*p)-8+8R(p-3) 



Prl p-1 v;: 

RCp) = X (2»*pM2*«i) = (p-1)(2**p)- £ 2»«i 

isi-v. • i=i 

= ((p-1)2*»p)-((2»«p)^2) - log n^(n+l) -n+1 

Thus if A(m) were close to the -median each time the /file is split, the number 
of- (Minparisons .done by QUICKSORT would - be of the -order (nlog n) . If all 
permutations of the input data -are assumed equally likely, then QUICKSORT does 
approximately 2nl6g n comparisons. " •■ - ■ "■ ™ 

'-• - *? • ... "... •"• •;; • . » ' . 

Space. Jlsagfi. At first glance it may seem that QUICKSORT is- an in-place sort. 
It_ is. not. . While the algorithm is working on one subfile, the beginning and 
ending indices (call them the borders) of all: the other subfiles yet to be 
sorted must saved on a stack, and the . size of the stack depends on the 
number of subsists into whicfrthe file ..will be split. Tnis,' 6f course,' 
depends r on n. In the worst case, PARTITION may.' split 'off one entry at a time 
in such a way that n pairs of borders are Stored : on the stack. Thus; 'the 
amount of space used by the stack is proportional to n. '■■ 

n 1000 2000 - 3000 4000 5000 - - 

HERSESORT. 500 ■ 1050 1650 - 2250 2900 

QUICKSORT. 400 . 850 1300 .1800 2300 - 

• *• (Time is in mill iseconds) •' . - -' 



figure 4.2.6-2 JERGESORT and QUICKSORT Comparison 



IfeSj&ng. The results of comparing QUICKSORT and MpRGESORT are reported in 
reference (4) and are sunnarized in figure 4.2. &-2. 

4.2.7. Effectiveness. Algorithm analysis has become an important part pf 
'.computer- science. The only issue that limits its effectiveness is that a ~\ 

particular analysis de p ends on a particul ar model of computation. rr jfcfa el 
assumptions of the model are inappropriate then the analysis- suffers. 7 

4.2.8. Applicability. An analysis of an algorithm ; can"" be limited by the 
current jstate of the art and the ingenuity of the analyst. 

. 4.2.9* Learning. Algorithm analysis .requires significant training in • 
' mathematics and computer science. Generally, it will be done by a specialist. 

_ ■ ■ * ■ ■ ' • -■" ■* 

4.2.10. Costs. The cost to analyze an algorithm is dependent on the 
complexity . of A : the algorithm and the amount of understanding! about algorithms 
of the same class. v . * 

';: ' . _ ■■■ \ • ? " v i , " ■■ '■' : ... :\ • . ■' ... . 
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• v*>3.2. •>,^ic :f eatures; -Die purpose: is to 'provide performance ^valuatioo anfc 
capacity- planning information on a system design. Hie process- follows the • top ; ' 
down approach to- design; through hierarchical, levels of resolution.' It can be 
applie d a t ea r l y ^esigh-stagesHifren^£M^ 

and where knowledge of their execution behavior may be imprecise. As the - 
.design- proceeds v-and the modules are further resolved, the estimates of their 
behavior and execution resource characterization become more precise, -The 
approach^ is' predicated on two- representational bases: on extended .execution 
graph models of programs and systems and on- extended queueing network models 
of computer' system hai*dware resources 

4. 3-3. Information -input. Ihe -information which is needed "for^bjs technique 
... ■. consists of functional design and; performance specifications asVf ollow's: '• ; ' 

a. Identification of the functional components of thefsoftware design \- 
to be modeled. — . • > . ■ 

. * • ■ ••• .- ■.. •. ' . ■- •• ' •• - ' • ; ,; . . • ■' v., .' 

b, . Identification ^of the execution characteristics . T (primarily, " 
• execution time estimate) of .each functional component.. 

• c. Ah execution flow graph which 'gives the definition^ of toe 'order of 
execution of the various func^nal components. \- ...... • . V 

d. " Execution environment specifications which can include information V 
such as operating system overhead and the workload on the system that could 
potentially Impact the particular software* underdevelopment. 

e. System execution scenarios whicTi provide the definitions of the 
: external inputs to the model needed for each simulation of the model'. ' 

. - *v Performance goals for the total --system- and components (an example 
is. an upper bound for the mean and variance of the Vesponse time for a 
specified execution environment -and scenario)^ ' * 

.- 4.3.4. Information output. Output from the technique 'will consist oft the '•' 
■' following:' •" "■ ■ "•• ■ «•,-•• V :.'■• : - 

- a. . A lower bound on the. perf oni^ce of the system. 

resu&s b * A " com P a f ison of the performance^ goal's with" the performance 

. - . .*•* - ; .• -. . • . . , ■ ■* • ..->. , -■ ■-■ . . :-.:::! ,'• , • ■ • 

- c. .Identification of. toe functional compcjnents whi^ greatest '•'■'•• 

effect on system performance. v > - ' > *•••'; „■ '.*''■ > 

. 4.3.5. Outline ^method.. Much of the effort in using this technique • comes * 
' in. -.the preparation of toe necessary input information. Once this has been ' 
' it is generally .submitted' tp a - computeY 1 ' which perfonns toe simulation of 
the execution of toe model, and reports toe results,' which are then analyzed , 
•vQQf? 0 ?* 1 revised as .necessary; The specific steps in the technique, are? f ,' 

'•* "" •'• ■ ■ ••* : *-" *--•- " : " ; "•' "' : ^r- -Vv ■■ ' •'• ."■ 
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. a w " '*' ".. ■•- ' .'' ' " ■ ; ''-V-v : '. ■ 

x a# structure of the. software design is characWized *^ of, 
^Its— -^umtiona^^ponents . In" ' Lhart,. -software' designs are gener ally" 
■ ; Z^SS^&&i in structure, a- model My: be; modified to represent the system at 
--different levels of detail, each- being analyzed at different stages in the 

z--. prOCeSS. — — — . - ■ — — ~-~ i — — — ^ — ~-~ r ~7~- — : ~ — - — * - 

, .' ■; • ; b. The order of execution of. the- components is . determined and the 
execution graph is constructed. 

■ • ■' ••• ••• ; ■ - •• •'• « -.- - •• ■• ■'. :. •• • • - .-' 

• c. Resource requirements . (e.g. , hardware or operating System 
resources) of the functional components are identified and a possible" 
environment - is studied with the specific resource workloads being determined: 
•These, workloads consist of ..the average wait and usage times for the resources 
controlled by the environment and used by the software (such as average disk 
access time). • ., 

' workloads- are then mapped into the model (as represented by 

• the , execution graph) - based upon the identified- environment resource 
.requirements of the individual functional components'. - ! 

\ e; Next, the system execution scenarios are constructed. The 

■ external ^inputs" comprising ; /each scenario may be formulated, for-example, in/, 
terms of the number of disk accesses required to find a needed data it'eP 
within. a particular component.. ' 

; . f.V Upon completion of the above steps, the model is 'driven, producing 
, system and component performance results. (The "driving" of the model is 
: usually done using a. system simulation tool such as GPSS. General Purpose 

Systems Simulator, on a coded-specification of the model.). ^ 

g. i-The performance results are new compared with the performance 
goals .of the system* If the ; goals are not met, performance critical 
components are then analyzed in order ; to determine where improvements / can be 
made. The design is modified and the technique repeated. This process 
continues until the performance is acceptable or until it can be' determined 
that the goals are unreasonable. 

4.3.6. Example. Finite element analysis is a .technique ' for determining 
charj^teristtcs;: such as deflections and stresses in a structure (i.e . 
building, airplane, etc.) otherwise too complex for closed form mathematical 
analysis. The structure is broken into a network of simple elements (beams 
shells, or, cubes depending on the geometry of the; structure) , each of which 
has stress and deflection characteristics defined by classical, theory. 

Determining the behavior of the "entire structure then becomes a . task of . 
solving- the resulting set of simultaneous equations for all elements. . ' 

The exampie develop, below is a portion of ,' a system which /does a finite 
element analysis. Consider the software execution* graph in Figure 4.3,6-1. 
Only the top level .of the processing is illustrated- here. The CPU time and * 
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I/O • retirements for each cbmpcinerit are shiqim In Table 4. 3.6-1 , 
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Figure 4.3. 6^1 Optimization Example (reference uD) 



Function 

Find ^beam definition ■ 
Sort on beam nunber 
Retrieve beam definition 
Find node locations 
Retrieve ^ node, locations 
Send data 



Total 



Disk Accesses 

,72 

.. 72- 

21 - •- 
36 ' - 

: o 

; 208 



CPU Time(ms) 
111 
32,644- . 
88,832 
3,018,726 
177,016 
2,600 



3,319,929 ms. 



TABLE 4.3.6-1 RESOURCE REQUIREMENTS - FOR OPTIMIZATION EXAMPLE 
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' t °^? nple ! e '? I/O operation is .assumed to.be 30 ms > ' ^Other 

specifications are unimportant in this example. • 

^!E e - We *^ -seconds 1 55.^' minutes) 

-'•ffif. •? .^ly "nacoeptable f or an interactive transaction, tte bottleneck 

analysis indicates that the CPU is the critical resource since it ha? ?u2£. 
: to; the elapsed time than the I/O ratio. SrtbeSS, JffnS Sde 

location" component is the critical component. - ' me , llna node 

The. processing details of this collapsed model are not shown; howeVer close 
JSgSfjL- vindicates that ;,a -finS^ta .K^ 9 2 

tolfecords m^^f? search^ keys, and then takes the intersection of 
S! Als©» it is found that the result of -the "find" for 
..^problem number search key is. invariant throughout the loop and need not ' 

tnl?mSkf \h?M^?I^ ^ e ^ Ure of Prohlem leads 2 tee SoSrSation 
res^^^ ^ V? 1 yields the same. 

■"ESS" «J! "find* ofav theiioite^ key from the previous pass throueh the 

SSk £i£ SS^E*?*. ^ r6SUltS * **, analysis .^caT^ 

ttese optimizations- are reflected in the execution ' graph • In Fleiie 4 3 6-?' 
r^t ^TxSti^' ^ "4^ SB^l « 
The response time has been redHSfid by 302?" 'seconds,', a. substantial saviiks! 

.gas?; ^S^^^ 

sSSs ^f^iSES* 1 ?' 11,6 ^ -B>e: resulting responseltaeVS? 
oT&,^ S if SSSS?" ^» continues unfil a resulting response tine 

definition ' \ |f { 

Sort, beam number 72 - * (il * 

Firid node location v . 4 ^'£7? -V ' v 

Reprieve beam- definition 72 fti'fta? • ; 

Fjlnd node-location:" ' B8,»32 /. : 

. B-tree I/O / 17 - mo •' /■ ■■• 

find « nodes . 2 • 44 ^ 

Retrieve 2 nodes . ' _ ' jS'SS > ^ 

Find 1 node ; ^ ' v 2'S° • / 

Retrieve 1 node . _ 

Record I/O ■ V 36 1 '25 ' ' 



Send data . < 0 

Total 



2,60b 

v 



, ." • , f \'< ; ' 2° 8 . : y'- _297,58a ms. ' 

TABLE 4.3.6^2 -RESOURCE REQUIREMENTS FOR REVISED OPTIMIZATION EXAMPLE 
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The perf onnance is still only marginally acceptable, but it is a dramatic 
improvement over the original designs The. bottlenecks are detected and 
corrected prior to actual coding and, therefore, the modifications require 
■ imal "effort. — — ^ — - — . . - — — • 
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Figure 4.3.6-2 Revised Optimization. Example (reference (1)) 



4.3.7. Effectiveness. - The accuracy of tfce perf onnance prediction is only: as 
• good as the quality of the performance specifications*, The quality of the 
^specifications, usually improves ' during the design process. A simplified 
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approach is used to analyze queueing network mod*?* ty>i« ™*>* < 

M of ^the bJ3LT^Sj5S^*«2Si "IKera? 

compensating features are used to .pffsett the apRna l ^MonT^T ' - 

^stributS'SSeS::, is senerany applicable.^ 

4,3.11. References. 

c - (1) SMITH j CO.,- "The Prediction and Evaluation of the Performance or 
: toalvsis (2 if S ^hi~" , nf? BR ?S' J - C -' "Performance Specifications and - 
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4.4.1. Name. Assertion Generation. 

4.4.2. Basic features* -Assertion generation is not so much a verification 
— technique-^ itself— as— i t 'is fotandational— tc-- a -varaety of other t echn iques 

Assertion generation is the process of capturing the intended functional 
-• Properties, of a program in a special notation (called the assertion language) 
- for insertion into the various levels of program specification, including the 
program source code. Other verification techniques utilize the embedded 
.assertions in the process of comparing the actual -functional. properties of the 
program with the intended properties. . / r ' •• - 

4.4.3. Information input* A "specification of the desired functional 
^^i"? 6 ? ' of J t ff P rc « ram is the input required for assertion generation. For i 
individual modules, this breaks down, at aaninimum, to a specification of the l 
conditions which are "assumed" true on a module entry and a specification of 

N-ttie .cono^tions^desired on module exit. If the specifications from which the 
assertions are 1 to- be derived include ^algorithmic detail, ..the specifications 
will indicate^cpnditions which are to hold -at 'intermediate points within the 
module as well : Additionally, assertions can state data characteristics, \e. g. 
loop invariants, physical, units, or a variable, as ^put onlyCcah not. be set) ^. 

4.4.4. Information output. The assertions, which are created from tfae^ 
functional or ... algorithmic specifications are expressed in a notation called 
the assertion language. This notation commonly includes: 1 -higher V level * 
expressive constructs that" . are found, for example, in the programming 
language. An example of such a construct is a setr Most cganohly 'the 
assertion language is equivalent, in expressive power to the first' order 
predicate calculus. Thus, -expressions such as- "forall i. in' set S, A[iJ 
Ali+1]» or "loens £xis£s .* aack-ahafc f (x) = 0" are possibles The assertions 
which are generated, expressing the functional properties of the program,- can 
then be used; as input to a dynamic assertion processor, a formal verification 
tech^ques th ^ UghS ' ^ cation sin^atxrs, ; and.inspect^^,^^amc4lg ot^er'wi: 

4;4.5. Outiine?.of : me^ hand-in-hand with 

the- . hierarchical elaboration of pr^am functions. ;When, during- development, 
a function , is identified as being needed, .it is usually.. If irst . specified by : 
what input r it; is- expected to take and what* the characteristics: of the output 
are (outputs are, of^ in tenns.pf-- the input quantities) v*-Eor-such a ^function 
it is possible to generate input and. output assertions without^any knowledge 
of how the function performs its task. The input assertion. .. . expresses the 
requirements on the: data^the function is to use during its processing." The 
.output assertion expresses what is to be* true on functioh-terndiiation. ^ 

Later, as the function is elaborated, ^the designer or coder wiil identify • the * 
necessary steps , to ; be ' taken, in order to accomplish what is required of the 
jfuiKition. After/ -eae^step it can be said ihat a- "part" .of the task has been>^ 
accomplished. That yart is necessary for the proper operation; of the next ^ 
step, and so on, until the entire ^function has t^ew-realized. The character ■ 
P f part::/c^ ^4cjca^^^ - an^ ^assertion> iii the same? way as the * 

description -of- the entire . function^ -The. output ' assertion^ for: one step - ''" 
represents ^ (at ^ least,- part ofl the input assertion for the following step.' f 
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Such assertions are called intermediate assertions. 

Each assertion,, input, output,, and intermediate is expressed using jtfae 
assertion language and is placed into the specification of the function being 
implemented at the appropriate 'points^ Thus, the program source text will 
include in it all the assertions developed during the - requirements, design, 



and coding phases. 



Seme programming languages infclude facilities for expressing assertions in the 
source code but. most do not. *^n such cases it is customary to include the 
assertions within comments, for indeed they are documentation expressing the 



aaoertions wi«ixn comment, i or. -indeed they are documentation expressing 
desired functional characteristics of toe\ program. Subsequent V&V tools, .such 
as dynamic assertion processors, are constructed to utilize these special 
contents: during their processing/ Dynamic assertion processors are able to 
checK the validity -of the source, assertions during program execution. Thus -a- 
metiiod f or dynamically verifying that the program is behaving . according, to its 
, intended specification is possiblei ' ... ■ . • *;& v : >•, •; 

For programs which contain loops (which is just about all programs), it is 
often- important to formulate assertions which are always true at specific 
points within the loops. . Such assertions are termed -invariant or ^ inductive- 
assertions. 




.4.4.6. •Example. Since assertion generation is so closely entwined with 
program development only a brief example is presented here. For more thorough 
examples see references (1-5). • . H 

During program development the requirement arises for sorting the elements of 
an agray or table. In' order to support flexible processing in the rest of the 
system, the array is declared with .a large, fixed, length. However, only a- 
portibn of the array has elements in it. The number of elements currently in 
the /array, when passed to the sort routine, is contained' in the first element * 
of l the array. The array is always to be-sorted in ascending*QPder. The 
sorted , array is returned to the calling program through the same formal 
parameter. r xW.r- • :. -..'V ■ ' -\ • v -. •; .•• 

' The first specification of the sort routine may appear as: 

'<:ij*-'-'-i . v SUBKWT3EVS0RT ( A, DIM ) - ' . ' ; ' 

•V.V.»..i • . .,; . . - . •• .■■ \ f *»t ••• .*%;. v • : »••'. •: \ ... i\- ■ 

C % ' . A is the c&ray to be sorted . 

C -. DIM is the dimension of A ' - • 

c .. r y,-; .• -■' •■ ■ '■. < >y v • 

C '•' sort array . - ' ;\. " • u - ' • ■ .. 

^RETURN. - ". . . . . "'.v... ■ 

•;. -••■^© -;::: ; A ; -.. ^ < (■■.. : ' . • v..-> 

: ... ; - Figure 4.4.6-1 Sort Specification/. ■ '. 'v.. 



•.';c 



t The characteristics of the subroutine nay be partially captured by the 
following assertions. Notationally, v= n or n and. island" * • 



. * 1 ASSERT OUTPUT (A(1 )=0 V A<1)=1 & true) v - 

. . . ' (A<1)>1 & FORALLI IN { 2: . AO)] A(I) A(I+1)) ~ ; 
\ ■ '■*■'.>.'' '.' ; ' <• V- • •••• ., . v •:' -.. .'}. \ 7 

The input' assertion notes the required, characteristics of AO ) and DIM.' The- 
output assertion indicates that if there were 0 or .1 elements in the. array, 
the array .is sorted by default. If there are at least 2 elements in the 
array, then the array is in ascending order. . _ 

-The next level ; >of -the program may have the following appearance, r An 
intermediate assertion "is .now shown. 

SUBROUTINE SORT (A^DIH) ^' : ~* m 'f- ^ 
C ." . ' J. • , \ ; ; 7" . .:■ •,' \ '"^f. ' .. . ■ 

C A is the array to be sorted - . s *V 

C DIM is the dimension of A ^ v>. 

ASSERT INPUT (0iA(i> DIM) , (DIM>2) - V 

IF (A(1) .LE. 1) GOTO 100 • 

ASSERT (2^(1 )*DIM) ^ V ' 

. C . .... . •• ■ - . :•• . ' •.. •.' .• . : •* \ 

■"C**"-- -.Sort rion^trivial .array - = .-- . ' • 

c • , -, .7 .7 . ••• • ■ ' 

100 ASSERT OUTPUT - (A(1)=0 v <A(1)=1 & true) V * , 

(A(1)>1 & FORALL I IN £ 2 . . ACT) J A<IKA(I+D) 

.. RETURN:-, , . • :•« •/.:' ; 

END 

Figure 4.4.6^ Sort Routine w with Assertions 




ii 



.Suppose a straight selection sort algorithm is chosen for the non-trivial case 
i-itive. ,« - find the smallest element and place it' in A(2) | fihct the next smallest 
V .arid place it in A(3) , and, so forth, where the original * contents 'j$ ^A(t) is 
exchanged with the element that belongs in the' 11^ position in -the sorted 
array); An^pDpj^Ia^e intermediate assertion is includetijfc&thin the sorting , 

C . .PERFORM STRAIGHT SELECTION SORT 

■';-> DO 50 J = 2, MM ;r ^ - ^ •'. .7. ' 7* 

v'V- : -...-f-^--. ■ : .'" - ;:-:-"^-^-V , 

C find-smallest element in r A<fi v v..;, A(A( 1 )+2) 

C . let that element be A(K) •^•'v ' : ^ 

C exchange A( J) and A(K) 

, p^: • v., ^ Cc^*:: S^c ; vv , r ' f ^ ' ^ , . ' 

ASSERT (2^C1)i r- : ; :,>:•'•• . -j - • 

(FORALL .I IN £2 . .' A(1)1' A(I)<LA.CI+D) ^ 
50 <X)NTINUE : 4 ^ . : • - 
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. A significant; issue which; we have, not dealt; with.^et is asserting, on 
; termination, that the sorted array; is a permutation of the original array. In 
Other words, we wish to assert that in the process of sorting, no ^elements 
•■ lost^Tp-do-this -at^the-higfa^-^elv-our first- .attempt: at the program 

... requires advanced. assertion language facilities. The interested reader is 
referred to references (t) and (5); ' • . v . 

HA.J. . Effectiveness. . Assertion generation, .particularly when used in 
. conjunction, wit* , allied -techniques like -dynamic assertion processing or 
. functional testing, can be extremely, effective ^ in . -. aiding V&V, Such 

effectiveness is only possible, however^ when the assertions, are used to 

• capture the important functional ^opert^sTof ' the program: Assertions such 
as the : following are of no use at all: T. ' - .* 

-t'^o'-:o-y'::^.y^ • • .. 1:= ■ . 

• i = ;v ■•. : , . : . ■ ••.•/■ : - ■ 

ASSERT IX) .". ; •' '.' . . \- .. 

;.. Capturing the important properties caft-be a difficult process jand is prone to 
- error. Such effort is well rewarded, though; by increased Ttoderstandirig of '* 

the problem to be solved. Indeed, assertion- generation is effective, because 
L*l ssert;ions are ^ ^ Parallel - to ?the pr^ogram .sp»ihcattora; r This . 

parallelism enables the detection of errors; but effort is. required; 

• ■•♦»•";••.'/.• •• ■".-.5 p * -:>.■ ■ 

A cost-effective procedure, therefore,'- is to ; develop i^tetmediate assertions 
only .for particularly important parts pf the. computa^orii -Input assertions 
snould always be employed,, arid, output assertions whenever, possible. c . 

r^Plijabnity. ^ .The. ...technique. -via ..- generally /applicable, in all 
development phases and for all programming languages. 

4.4.9;' Learning. Training and experience". in writing assertions is the key to 
their effective use. Thoughtful consideration of the material contained in 
> the references- should enable a programmer to begin with useful assertions. 
Experience will sharpen • the "ability* " especially if a dynamic assertion 
processor or other allied technique is also used. . 

4.4.10. Costs. Assertion generation is generally a manual technique, i.e. , 
no machine resources are required. Effective use requires . thoughtful problem 
and solution consideration, 'but no more than L- is • normally required ' -in 
professional task performance. Tools;<io exist that use symbolic execution to ' 
automatically generate loop ^invariant assertions. The cost then becomes that 
of symbolic execution, • — - ' •••• 

'4.4. TJw References. ■■,,-,..,»..'*••;- : " 

• (1> TAYLOR, R.N., "Assertions .in Prc^amming Languages" SItiPLAN 

* "~V 15,: 1, January vi9&^, *pp. . 105r114. ~ T : 

j^Ol^^vv^HtflpS^:;- Ri l i ; ?ThS Logie 'of ^r-CSooputer • Programming* ' 
. JBB&lEE^ a^*;3r^nt^"»4 it 199^2^ (especially page^l 9S£3p4)> •' : // 



tisSZss&, Vol. 



. "flit's % < •/•'.'■^ • ~t .• ■. 
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(3) HQARE, C.A.R. , "Proof • of Programs • FIND" CACM .V. -14; * 1. 
January 1971, pp. 39-45. ; ' l~~r~ - ? ' \...'&'. 



pages 7-10,, '17-28, '57-72. 

• (5) CHOW, T. .S. , n A Generalized Assertion Language", Proceeding s of 
Jjte ■ SfififlDd International Conference on' Software Engineering ,£an Francisco,; 
. California,; pp. 392-399. ^* " ' 

(6) STUCKIj.L. : G., and, FOSHfeE, G. L. , *New Assertion Concepts for 
Self-Metric Software";.* Proceefllnfe • 'flf. 1975 Conference on Reliable 

Software, pp. 59-71.. 

t,. 



. * 




. ' 5 



■7 T . 

.*• - ■ ■ ■ • ' 7 .< K\ v ■«* -. 
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.,4>5 r 2. Basic ^eafores. Assertion processing * is the- prbcgss whereby the 
|a?graD*?^af&^^ ^-tiescribeti in the 

- Previous^ section) * are * checked during pr^am; /execution. As such, 1 the 
techniques serve as a ^ridgfe.BelsreA- :1lift'Bbi^ieioitt^j^ofra correctness proof i 

4.5.3. Inforinatton Ihpu^^^ 

, Prograar whic* . ^contains^ the^ -be, pro&ss^ ^ e pr c^am can be 

written in any* language ttafemay be. -restricted to a parftlifar language - if an 
* autanatic tool ,ia used to perform the dynamic" assertion processing- Moreover, 
if a tool is used, the fbrmatsfor^ specifying the assertions will be that 
• defined by the particular -tool. Generally* * assertions are specified as 
comments in the source program. • 7 - . : • . - / 

415.4. Information Output.. Output tfrflfo a' dynamic assertion process normally 
consists,? of a^fst of^e assertion checks which were performed and a list of 

, exception conditions with trace information for determining the .nature- and 
location ^f the yioiatabns^ - ■i.'C;'-' v :<■■■'■ V 'h 

_ • . y •:-.:•.••••-•?>' ; # : v "■' \- ,r i>- .v.".;' 

«-' 4*5-»5. ; : ; putline^ pf JWethod*; \ lhe. ass^tiohs f are generated by the developer as ■ 
described in the "Assertion .Generatioh n technique' in the previous section. • 
The assertions are; then .translated into host language .program statements which - 
actually perform the . assertion checking, at program . execution timei- The 
translation can be; done manually , or through the use of ~aii "autcmated dynamic ' 
; . ^assertion processor-. ,.. . . ~- ■,. . v , . v "'• ■ ;•• v - ■/...•-*••" ' 



The t^^\^6^^ogG^~xi '^(mi ixi the fcllowiijg illustration. An assertion 
of . the.f orm: . » ':. '.*.-•". • * ' - ■ - ■•- 



(* ASSERT: condition*) ' " ■ . •* . : ^ ,; \ ( ^ ' 

is translated, int 



IF NOT. (cooditiin) THEN' Y ^ ■ " \ ^ 



I¥octesa' assertion violation; . *- r . *- J 

; The pro^ minimally, ~ keep track of the 

total number of violations for each assertion, print a mes^e .indicating that 

, a violation of the assertion has . occurred, ■ and print the values of the 
variables "referenced Via the Cassertiom In addition, the location, i.e. 
stateneht number, and the numoer {pf tifl«s the assertion is <±ecked^may' be kept 

• and printed when a violation ofccurs. iiJr*<-^- V- : / -<^-: •' 

Sufficient' information shouldi>e reported upon violation of ^^n^ assertion to 
assist the programmer ,of ,the specif^c' nature of the error.' ' • ' 

autxaiated dynamic ^sertioh. processor can be of great assistance by 
an^iating ; for the programmer, the burden of hand generating the source code 
necessary to perform .the assertion checking. Not only will this save time but " 



. It will also; perform the' translation more reliably-. £ . •*'£'•• 

,^^ifyiiig i assertions a valuable form c>f - docunentation and 

j-^also^^ 

• ^ is important to note that- dynamic assertion. processing * for non-real time 
programs^ must • not alter: the functional behavior of *a program, i Use of a -good 
justed 

• the ;ampunt. T Of, which, will depend on. the, number of assertions which^re 

^ter .toe functional hehavior of a program by altering the execution tin£ig. 

l*a ofder^tO effecti^^^ ..processing, .test data should be 

generated which-will cause the execution of each assertion. / . ' V:^;-- " 

; i&5 # ^S?^ 6, P 16 P r °gran segment in Figure 4.5.6-1 is taken from a Pascal 
program which calls on .routine 'sort' to sort array 'A', consisting : of 'N« 
integer elemente, in ascending order. The assertion following the call to* 

" S^^rlPSSJS 8 * ^l^^ts are,inaeed,in ascending or4er upon return from 

• origSa " " left ^ 016 lin e' number ^frcm the • 



12 lac , .... ; 

T3 ' - ,v N : integer; - : ; >: ' 
14 * V A. : aoayjl .. .MAXNJ o£ integer: 



;"26 , f ; bjBgixi,v, v ,,, v 



■t V 



v56- ;, ■ . sort (N, A) ; . '^X. ' >■ ;." " C 

^ Figure 4.5.6-1 Source Program with Untranslated Assertion 

The program segment- in Figure 4.5.6-2 is -that which results -after all of the 
assertions , have been translated into Pascal. Note that a rather large number 
'^SlST^'lff^ "^ t 0 dement toe assertion; This is due to the rather 
SS^^'?!??^ if" 1 "^ to implement an "as^ jiir^U. . ' .".•/Simpler. 
..a^sertions^will require fewer statements.. The spec could be reduced; -though " 
•toe use of a common assertion violation procedure. - * 
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43 ' ■_ N-;- integer; - - \ ■; /v , ' ' ■> . . -.. , - 



; t4 ' . : ati"& [1.;MAXN] ^integer; ^ -,':V 

15 *' .»' f - Nunof Asserts] ^integer ; 

..." * t AssertXiqtCpunt :~anai It..: Numof Asserts] *s&. integer; 
17' / \ assert : boolean; : : v>&.$& 0.:: ;.v:' : - :: . { ■"- ' 



77 •'/ .- , sort (If, A);.-" . ""'^ V V : / : ' : '■ V /H" : * 4 

78 ; a assgri fojall: in [1 . iN-1 ] :' A[i] =A[i+1 ] «) ; v 

79 ? AssertXqtCoiant[33:= Assert^'tCount[3]+1> - - 
80" . . ; ■> assert true; . ■.' *■ v "'*.•.•' •• 

81 - •i^.-s./t...;,..':--:; ;-v.,!v : ;,;',-:;; : . - . V';.., 

82 H0ilfi(i<= N) aadCasser't) daC* check assertion *) 

83 •• . : if. A[ij>A[I+1> ibso.. v : * " - • •; -i; '. • 
"8* . . > . ^ assert := false t'-v"."-' '.' •'. '• 

85. . .'.?" * jglse:'**- v ' ; ;' • V * • •* . ..•""« .' ■ 

86 v '~ : "-\ • ' ;: ;i := i.+ 7Tr-r : . .' : .. '■ . ' - .. « V 

87 i£ nj2t ^assert Jfcbjai JjsglttC* assertion violation"^ 

88 AssertYioCount[3] ^ AssertVioCouht[3] = 1; ' 

89 . > Writelh ('violation of assertion 3 at statement' 57^0 J ••••^ 

90 . . . - Writeln ( 'on, execution: ' , AssertXqtCount[3] ) ; ' 

91 " , . -> ' - .Writeln .( 'arrayA •= ;* , A) > ' \ • ' . . 

92 .v.:. eui t« assertion violation . *)/, " . '- • ■ .. 



Figure 4.5.6-2 ' , ■Source Program with Translated Assertions 



During toe testing tiie following values" of A were used in -successive/ 
executions of the sort routine. ~ " ... 



execution ; . v - " anay. A 



\ 



A > 



"•^.i ;J > ; 0 1 - 12; ^7 "S » 171 201 251 390 50V 
;^V2;^ 0 % r 3 -53 /' h 201 %1 390 251 50V . 

3 ^ ? 501 390 ^5l/201 171 53> '27 12 3 7r "b 

^ . " ; : :; : ^o 6. o o o o^C^'fi^^- , 

\ • 5*-v f " " t- " 0 0 0-v 10d J00 ^ 100 999^. 999 999 1000 
resulting execution produced* the following assertion violation: 



« -v. 
■'.> • 
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?>•• :,v : ..• ^ . ;r^V'***,-: •". ^ •-• 



. violation or assertion ^ * \ f -V 

£' arrqy fe,, 27 £532i3IlLj01>^ ^34q v 



.This was thfe only viola . * * ■ / ;. 

Subsequent -ahaiyMs ^ error "was" due y tor 

an ^ddr^rC«e^^ s *v ;■ ^ 



r #3.7. Effectiveness,; Ihe ef f ec#ven^^ 

%de^ of the> desertions ;in<£u&^ 

analyzed. Hc«reover/ jX; v t^^ is being *done by hand£ the ^amount - 1 of v . 

time required ^ 

pi$c^.:>^ the technique can be of >■ 

f : significant value* in revealing the presence of prpgram e^rorS; * " r >i - v 

' . ' •.-"1 ~ . - *. ■' **. r '"-■"Tv " ^ '" " - ' • * C — ' ' ' ' ••■».-', ■ ■ 

; 4.5.fc ^plicabilitjf^ i^ gOTer^iy^applidaa ' i _ ^ 

: 4.$.9. Le^ 'is ' all that is - 

necessary in order vtb If a tool "is used, then a^ 

hdar or So • SiojiLd be .si^icient^ ^tO;. learn - the specificaticxi^ syntax for * 
asssertions ^ acceptable v to ^ tfe 

'assertions (see order f $r this 

; 4.5ll0v otosts*." .The cbs entirely ^ - 
^ cdnpfrised of jthfe amount :'"> pf tiro required .to t^slate the assertions iritqv 

source code* If "done ^ m^iialiy y thi? could anount 'to si^ ' If 

v djcme v aUtcmtifciaily ,^ the^ "•cost will be on tihe.or4er of collation (Assertion 
*• Processors ar^e usually as. source lianguage pr^rocessors) ; If a 

"tool is ^ not available, it m^;we^ cost to develop one in-house, > 

4.5*li • ; R^erences* r 5 ; v ; N ^ . ./ " , >■ ^ i' ■ C^vo^ ^ J"' V ;C . 

(1) STUCJCE, L.. G.,, and POSEE,: d.- L M 4^W Assertion • Concepts- -fc?r r 
, Self-Metric t Software" T Proceedings; : 1Q7»? Conference jsn , Reliable 



v' (2) ANDREWS, D: M. , "Using .Executable 'Assertions for. Testing", Jith iDIUJal 
Asilcniar Conference' on Circuits r " Svsteaas/ ^d DeviGes.^Mov.y -1Q7Q> 
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- v Ba ?^ c -^ eat S pes « * ^u«i-efffect graphing a' test case design 
nettodology.- ^tt is used 'to select in a systematic manner a set.of testcases 
, : a high probability of detecting errors that: exist in a program. 

^ - This technique,;, explores^thfczinputs and combinations^^af^putHren^ 

progran in developing t^t. cases.< It is totally unconcerned with the internal 
' j£rJir < 0r of ;a. Program. & addition, 4br each test case deriled, 
the technique identifies, the expected outputs. Hie inputs and outputs of 'the 
V[<>er2n are determined^ thrpu^ . analysis of the requirement "specifications. . • 
These specifications are then translated %intg--. a Boolean logic^network or 
S^sis^ * network is ; used ^t^ .der|ve. test cases for the software under 

4.6.3? ^ Information inpufe^ the ihf;ormartion that is required as input to carry 
out this^t«^que is a -natural la^igu^ 0 f the program that is 

to be tested. ^The specification should include all expected inputs and 
combinations of expected inputs to the program, as well as-expected outputs. 

4.6.4. Information output. The information output by the process^ of 
cause-effect graphing consists of the following: . , . : r; 

a « An identificatiori of ffio^ete ;br'4feonsistent statements^ in "the 
requirement specifications. .. . ; " ; 

b. A set of input condjLtioliJ^n(^the software (causes) . ." \.[ ]\. 

" ; c - A set of output conditions oV the software (effeqts).! 

."■•'*'."•' r ■ ... * ■■ Wh'y' . -. #yJ5> ■ ' * ' 

conditions * & ^ ^SS^ : ^ e ^ Pl *; conditio^ to the^ output 

£V*^0^*M^2* stable that determines which input 
conditions will result in. each .idenMfied output condi^tbh. : 

A set of test cases; a . J- '. • V;r • . • 



. g. The expected progr^m^sults for each derived iest case. 

The above outputs represent - {t&V* ftsmt - of perfprmii^ i the ? varidus steps 
recommended in cause-effect^grapMng. -:v.' v ".v. ' ♦ - 

4^6.5. Outline of : method, "Sj^^se-effects^grapJ/ls a formal language 
translated^ from a na^^^languaige specif icatioKv The graph itself is 
represented as a ccmbinSt^ala^c > network. ' Tfce , process of creating a° 
cause-reffect graph to deriyeit^ upases is described briefly below. 

• • '' ' /&-:?&^>$^:HtZ*t ,: ^'i->ie':- • " : - :*" ' •• 

. a^ Identify^!. Je4^e|fents of : thfe system and .divide them into 
separate identifiable^ti$i«?i?- .:) ■-■ 

;• :;.t'^^^ ; v^pv •' v'^-v ' •. . f . y % *. , 
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; b. Carefully analyze the, requirements to Identify all the causes arid 
effects- in the specification. A . cause is a distinct- input condition; an 
; effect lis an output condition- or ..system . transformation (an effect that an 
.input has on the state.„of the. program or .system) . »'..' ■'■■■■< : : . • 



- c. -Assign each- cause-andr^fe<^a--uniqu^^ 



ERIC 



%- -Analyze . the semantic content of the: specification and transform it 
into ^ BcH^l^ean. graph linking the causes and; effects; this is the cause-effect 



^:^*iR€5^i^>jeaieh ot|U8e^a^<^Mfc^^^no^ identified byVits urcique • 
number. ...... ' . 

•>•'.;. o List all the cause nodes vertically on &e ief t side of a sheet of paper; 1 
list the effect nodes on the right side. . - 4 , ^ 

o Interconnect the cause and effect nodes by analyzing the semantic v -..-$3 
content of the specification;. Each cause and effect can be in one of • . 
two states: true or falser ; Using Boolean, logic*, set the possible 'states 
of- the causes and determtne;.under what s o<3r^tions. each effect will 
, be present. •. 'v ' - .••/.:•.• . 

6 Annotate toe<graj4i with: constraints describing combinations of 
■■' causes and/or effects that are impossible because of syntactical or 
'- Environmental constraints, - ■ . 

- e. By methodically tracing: state 3 conditions in tfiferjgraph, convert the 
graphC into a limited . entry decision table as follows . For Each effect , > trace 
back-through the graph to find all combinations of causes .that will set the 
effect - to : b^, ytxife. Each/such combination is represented as a column in the 
•:deeiAon.t^eV;;lhe sfate; of all other effects should also be determined for 
each such combina tion^ Each' colunn in the table- represents a test caSe. 



f. Convert the columns in the; decision tabid Into test cases* 



*; Thiaf tec^que to create test cases has not yet been ; totally automated. 
However, conversion ? of the graph to the decision table, the most difficult 
aspect* of the technique, is an algorithmic -process which could be automated by 
a computer program. "\- 

4.6.6. Example. A database management system requires that each file in the 
database; have its name listed in, a i master index which identifies the location 
of each: file. " "The index is divided into ten sections. A small system is 
being -5 developed which - will allow, the user to' interactively enter a command to 
display any section of the index at his terminal. • Cause-effect graphing is 
used, to develop a set of test cases for the system. 

. a. /Ihe specification ^ 
To display oneTof the ten possible index sections, a command must be entered 
. consisting of a letter and a. digit. The first character entered must be a D 
(for display) or an L (for list) and it. must be in column 1 . The second 
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character enterfed must be a -digit ;^0-9) ; in colunn 2 . If ■ this .command occurs, 
: tfie iridex section identified by the digit is displayed on the terminal • ; If 
the first "character is incorrect, error message A is printed. the-secbnd. 
character is incorrect, error - message B is printed. The error messages are: 



A: INVALID COMHAND ; ^ ' 

; " : B: ; BJVALlD DTOEXrN 

bt ; The causes ^d ^Cf ecte. have been identified *as f oilows * 
been assigned % unique nunber . . : - 



Ea?h has 




T. .Character in column 1 is D. 
2; Qiaracter i^ colunn -1 is L. * 
3. • Character in colunn 2- is a digit. 

Effects . 

50. Indfex section is displayed. 

51 . Error message A is displayed. 

52. Error message B is displayed. 



c. Figure 4:6.6-1, a Boolean^ graph, is. constructed 
of the semantic .content of the specification; . * ± 



through, analysis 




V ror 



A sand 



f* = not 



Graph 

Ncxie^20 is an intermediat^iid^e representing the Boolean state of node 1 or 
node* 2. The state of node 50 is true if ; the state of nodes 20 and 3 are both 
true. The state of node 20 is true if the state of node 1 or node 2 is true. 
The, state*, of node^ST is true if the state of node 20 is not true. The state 
of nolle 52 is true if the state of node 3 is not true* 

Nodes 1 and > 2 / are also Rotated with a constraint that states . that causes 1 
and 2 cannot be true ^imult^neajs^ (the Exclusive constraint) . 

-•v./lL The grajAi is donvertea into a decision table, figure 4 ;6. 6-2. For 
each" test case, the bottom of the table indicates which effect will : be present 
(indicated by a 1). For each effect, all combinations of "causes , "-that will 
result ; in the presence of the ef feet is represented by the entries in the 
colushs of the table/ Blanks in the table mean that, the state of the cause is 
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-- Causes. ' 


i 


2 








-.1. • 


0 


.6.' •.. 




:>.. -.2' . 


• 0 


•U. 


0 




Effects- -V . 


. 






0 




1 


1 


0 


o- ■ ■ 




'..;0.;" ; 


0 


1 






0 


.0 


0 


1 



Figure 4^6 .6-£- I^ision T^ble 



e Each colunn in toe decision ^^te^ test 
figure 4. 6>6r3. 



cases/ 



Teat foaff j t Inputs 
3 B2 



: Expected Jtesjifcg 7 
Index .section 5 is displayed 
Index> section 4 is displayed 
INVALID "COMMAND - ... 

INVALID INDEX NUMBER 



. . , ' 4 Figure 4.6.6-3 Test Cases . ' - 

Ef ^ ectiveness - Cause-effect) graphing is a technique used to produce a 
useful- set of test cases. It also has the added capability of pointing out 
incompleteness and; ambiguities in the requireaient specification. • However, 
Jh^ technique does~:not produce all the useful test cases that can he 
identified. It also does not adequately explore boundary conditions. 

4.6.8. ' Applicabnity. Cause-effect graphing can be apM generate test 
cases^ in^any type of computing , application where the specification is clearly 
stated and combinations of input conditions can be identified. Manual 
application of this technique is a somewhat tedious/ long, and moderately 
complex process. . However, '.the technique could be applied to selected modules 
«jere complex conditional logic must be tested. 
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.4.6.9; learning* „ Cause-effect grayling is . a 
that requires sane knowledge ctf Boolean. logic; : The requirement specification 
/of. the system; must ralso be ^clearly understood in order t*>= successfully carry, 
out the process. =>; ^ -V ~X-iS£i^ : :--'-< '-v u V - i ." ' • 



4767T0. Costs; Manual application of" this technique will be - highly labor 
intensive. -• ~ " '• .•-.»'--/ . ••• -V 



4.^1. .References. 



(1) ELMENDORF, WtRV, "Cau^Effect Graphs in Functional. Testing, " SH 
Systems Development Division, IB^je^Sfi^jPoughkeepsiej New York, 1973. : 'V 



■ ' ..Jr. 



X2) MYERS, GLENFORD* V "The - Art of Software^ . :v>Testing; " ;• 
Wiley-Interscience, New York, .1975^-\_ ... .'• w? . • ■ 

> (3) MYERS, ' GLENFORD, , "Software' Reliability: Principles arid 
Practices," Wiley-Interscience, New York, 1976. 
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A.7.1V Rame. {^Auditor 



4.7*2.' Basic f eatures J : code auditor is -a canputef program which is used to 
examine 'sdarce^ <^ land automatically determines whether prbscribed 



4,T.3.^ formation input. Ihe i^prnatt^ a : : qxie" ; auditor ■ .i£ - the 

source- code / to be analyzed and tjbe ; commands, nebessary for ta^ 
opei^Udn. " Vv" • *V ' ' " ^ f ' . ■ ^; 

4.7.4. >Iitfd!TOtion outputs The infonratiOT is output by a code auditor 
is a deteimEhation of whetfc^ the ccide feihg afiai^ied adh^es to prescribe 
programming ; st^ detailing 
which standards ■« have b^ -" violated and where the violaid,ons^^ur. This 
information can appear as Vror messes 

a separate ^report. Oth^ I diagnostic information, such as a' cross-reference . 
listing, inay also be output as-an aid in making the needed corrections. : 

4.7.5*, Outline of method. Code' auditors are fully au&a»ted v tbals which 
provide an objective, reliahle means of verging tha prd£ram complies with 
a specified, set of coding standards. Seme corn^n programming convfentionsv^that \ 
cpde auditors- can check for are given below. N - : . . x 

Correct syntax ^/po jall program statements confora;t»-tife ^^ifiqatiSns of 



the language de£initioij? 



o Portability - Is the; pode Written so that it can" easily^ ^ on; v 
different computer j^^i^ai^ons? ,„ -V- •V?!'::'v\\'.-.: • 

o JJse of structured programming constructs - Does the code ifiake : prdper use 
6t a specified set of coding constructs "such as . IF-1HEN-ELSE or DO-HHILE? 

O.Size - Is the leAgth of any program unit not more than a specified nunber 
of statements? V' ,S V' 

o Ccmmehtary - Is eafch program unitg&propriately dociicented;- e.g., is each 
unit preceded r by a blbttk. of ccnmaTO^ch, indicates the function of the 
united tto^ ieach variable used? 

o Naming conventions - Do the names of all variables* routines, and other 
symbolic entities follow prescribed naming conventions? 

o Statement labeling - fioes the ^nimeria labeling of statement/ follow an 
. ascending sequence throughout each program unit? : ; ., - 

© Statement ordering - Pp all statements appear in a -prescribed order; e.gi> 
. ; :ln a; Fortran .program, do all FORMAT, statements appear; at the end and; DATA 
statements before the first execu^ble statement of a roiitine?*^ 

o- Statement format - Do &1 statements follow a pres^ibed set of formatting 

rules iihich improve program clarity? e. gi, are all DCMffllLE loops 
" approiriately_ indented? 
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'^.f? <*fBpwtrated by. .this list, code auditors vary in sophistication according 
to ^ t^rafunction, Ea^ auditor, however, requires sane form 'of syntax 
- ana^s to be performed. an 

inter^ suitable for analysis.' Because : this type 'of 

, Processii& is fouh^^ 

. of a more general tool' having many; capabilities. For example, a compiler is a 



a 




form of code auditor that checks, for adherence to the! specif icatidns of a 
- language -definition. EFORT, a tool used to checfi Fortran programs for 
adherence to a portable^subset of American National Standard Institute (ANSI) 
; Fortran: 66, also has 5 the capability of generating a cross-reference listing. 

Code auditors are useful .to^ programmers as a -means ^of ' self-checking their 
.routines prior .. to turnover .foe integration testing. These tools are also of 
value to software product assurance personnel during, integration test j 
prior , to formal validation^ customer "delivery./ 

4.7.6. Example^ 

: y ^. Application; A flight control program is to be coded entirely in 
vPFORT,- a portable subset of ANSI Fortran 66. The program^ is to pe delivered 

to a military government ^agency, which will install the. software on various 

computer installations.- ;In addition;: : -the customer requires that each routine 
. in <ttoe program be clearly documented in a a pr escribed format. r All internal 

program comments are to b6 later compiled as a,. -separate source of 

documentation . f or the programi^ r, ... 

, D ,* Error. A named bciamon block occurs in several routines in the 
program. In one routine, the'def inition of a variable in that block has been 
omitted because .the variable is not referenced in that routine. This is; 
however, a violation of -a rule/define^: in PEORT, which requires that the total 
length of a named common block agree in all; occurrences: of that block. 

c. Error discovery. A opde auditor which checks Fortran for 
adherence to PFORT detects this error immediately. The programmer of this 
routine is informed. that the routine is tp be appropriately modified and that 
any confusion over the use of the variable is to be clarified in the block of 
comments that describe, the function of each; defined variable in - the routine 
A -code "auditor. ;that -checks for the, presence of appropriate comments in each 
routine is used toiverify that the usef of the variable is appropriately 
documented - . At- the end of codg .construction, ; m such internal program 
documentation will . be colla'ted and summarized by another 'code auditor which 
processes machine readable documentation imbedded in source code. 

4.7.7* Effectiveness. Code auditors are very effective tools in certifying- 

that software routines have been, coded ^ : in accordance with prescribed": 

standards. -They are much more reliable than manually performed code audits 

and are highly, cost effective as they are. less time consuming than manual 
.audits. •■" j ■ . .« , ... _ : j '-. ■ ^ • 

4.7.8. Applicability.-. Code auditors can be generally applied to any type of 
source code. However, each specific tool will be language dependent (i.e. , 
will operate correctly 'only for specified source— languages) , -and V will only 

■■ - ..." • . V . V - ' 
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' isl6< ^^^^ appe^'in a^rescribed'fb^t. . V- .•' •"■ 

. ^'^1 V^ning.. iJtoispecial. traifl^ is -required to use cc& auktors ^ 
o^tf 2?°^ a wide viarie^ofcpeople (pr^nSsT^agerf V 

!Su^?^^^ , T mel> f^ tdners) ' ^se lri' their us^iT^^tart 
In order to use code auditors effectively, however Stae ??arn?n? 

auditors are generally very inexpensive to use as their 
, v overhead is usually no more than the .cost of a compilation. . . 

4.7. 11. Refjprences. "•• . ,:.>'* . - . . ; " ., v : • -. „ •-' 

v^rW* }< ^^ to the 

* t3) ,S tHE ?' ' User's Manual for Code Auditor,. Code Optimize* 

- ' ' (4) HOPKINS, T: R. , "PBASIC- A Verifier- for BASIC, " /ScJWe Practice 

ml EwlencfH vol. ip, pp. 175-181, 1980.:, . ; ,.? ffl f^ f 
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|l ? 8.1i F*i^«: Comparators.^ ? £ 



4.&2. Basic featwes^ A comparator is a. coppmer :program usebV to compare 
: two versions ; of source b^ta .to establish that, the two versions are identical \. i 
or to speciflcallyilfert versionsVocCur . , e 

. Ihfonnattoh ^ two - Versions of >- 

>>source "• data to fe.cbmpared.and toc^^bcainan^ .necessary for the comparator to 

I operate. The source, date n^ te: .. - r r * . * ' 

• : * >'•••';«:>.«• '-V. * -a*-. Source .pnoj^ams . «>|^-^7'f - • v: "' : - • 

: "',.-"/i>-;h< Sets <cf program test cases* or results •" --•'"•v. • 

". «, -••:«Cf • Databases v ••>-.•<■•.-;> ^..^ ''W :!> ' ■ ■: . ' • w .• 

... ■ "d. ^bitrary data' files, v. • 3£'£.« ■ ■ •' . V\V' ; ''. ' 

Many comparators provide various user options, 4 such aV whether blank lines are 
•to be included in compare processing, to control comparison operation. 

•• > ■ ' • • . * * "» '■■ <•''''. - '• ■ ' 

.,v 4.8.4. ...Information-: output; Ihe output? from. a. comparator is a listing of the 
differences j " if \ any, between- the two versions of input. Various report. . ... 
writing options are usually supplied • by., the comparatcc;!- to designate toe 
desired format of. . the % output^s£g4f f whether each difference found should be 
preceded . by tlihe numbers. Many general, comparator utility 'programs installed 

** in large text-editing systems can also create a file of text-editor directives 
that Can be used to convert one input f He into toe otoer." v; : \ 

4.8.5. Outline of method. Comparators ar^ automated itools which serve .. 

to eliminate vthe:tedLous, tfene-consuning task of performing large numbers, of 

comparisons. ^Hxey-are most: useful during program development and maintenance. . 

During" program 'development, toey prbv^ of ensuring that only the 

j .intended portions of a program are changed wh'ehjikxiif ications are to^ be made 
•- to the latest* Version. When regression testing must be performed ^following 

"software corrections or ii^t^ t y '6q^mtm prcwideySari; efficient "means of . 

comparing current test- cases and; test results with past ones. 



* Comparators are widely/available and are often provided as general utilities 
in operating systems. Other comparators may be more , specialized and require ' 
inputs files, to ^ ctf a prescribed format in order for the tool to operate 
, cori^tlx; ^ : * ; V V"..'' 

Oomparators are invaluable txx&s in assisting configuration man^ement and 9 
change control as the software takes different forms during development. 

4.8.6. Example. ' • . J- : ;f.^*A-- . . . " : ; 

a. Application. A. large command and control flight' ^ ,.. 
lis - being developed. During system testing, the gjraeratiofe^ man^ different^ 
databases is required as a source of input data for ■ each associated test case. 
Strict control of ' the databases, ; including -i^htatfication of their : 
similarities andr differences, must constantly be mainta^ed in order tot 
properly verify test, results. £ : 
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• b. Error. A:bug in the ^tware isau^s^juie-execution of Test Case 3 
\ , to generate, test results which are totally incompatible .with the results o£ : ^ 
! -- Test Case >1,vtbougb:^ identical* 

'^wr, c; Eriyr. -discovery . A A comparator was used to compare thfe. ^ databases ' 
: used in Test Case 1 and 3* The location of specific differences in ; the- two ' 
files determined exactly which inpulr data should be examined .more closely and 
when traced through tiae program the error was found. 



4.8.7. Effectiveness. Comparators are most effective .during software testing 
and maintenance, when periodic modifications to the software are anticipated. 
Their overall effectiveness is dependent upon the quality of their use. 

4.8.8. . .Applicability; ^ Thisimetho* - .v V 

4.8.9. Learning. A minimal amount -of effort is requir^ito leara how tx) use 
comparators effectively. T^; : \.jti^<sCf.^et'~ documentatibn should provide 
sufficient info^tibh^or ite proper utilization. ' .;* •- 

4.8.10. Costs, axflparat^r^tar^generally inexpensive to use. Their cost is" 
similar to f3^=cfrpiktprn^}^\pea^ of read operations on one file. *'V 

4.8.^1:1.. ;J^WOfc^.> 1 ' .^.r;^; •;; ,\ '' ; ' ■ ;. ' -?) . X '.' .• 

C1)V v HETZEL^ : William, - "Program Test Methods", ^Prentice-Hall, Inc: , 

1973. '«/>'•/•'■'•■ ■-. •<• , • ' ■ .. r . ' 

(2) . DEC IAS/RSX-11 "Utilities Procedure Jfanual", Digifcgl Equipment 
Corooration T '1978. 
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■ • \;4.9^'. ■ Wam^i * Control Structure Analyzer; ' . ' : '- T '' ;.'.'"> • : . •' ^ 

4.9.2. -Basic features. . 'Application of an .automated structure -analyzer to 
n. either .code ;pr . design allows detection of sane types , of improper subprogram^ 
L ±u^&*n&Yicl ggggK^ also -identifies- Control 

branches and paths used by test coverage analyzers: - A structure analyzer is .* 
-also useful iir providing required input to data flow analyzers and is related * - A 
in principle to code auditors. . ; ' " 

4 t9«3» ^Information input . s ' Two input items, "are required.- by . a structure" 1 :; 

* analyzer^ Mhe first is the text of the/program ^tdesigh to be analyzed. 
TypicaHy the\text is to be provided; rta^.thV^alyier in an intermediate form, 
i.e., after ^canning ^va^d 4 i^rj^nj^^r biuft .not as ^iec^ -<»Sde.- Often structure 

The second input item is "a specification- of the control flow standards to be 
r checked. Theses standards/ are often completely implicit in that they: -may be 
part of the rules for programming in the given language^-.or design Wtetion. 
An example of such a rule is that subprograms may not be^called recursively' in 
FORTRAN. • Individual projects may,' however,' establish additional rules •for 

- internal use. Many such rules, for instance limiting the number of lines 
allowed in a subprogram, can be checked, by a code auditor. Others, however, 
can require a slightly more sophisticated analysis and are therefore performed 
by a structural analyzer. Two examples in this' "category are "All ' control 
structures;, must be well nested"? and "Backward jumps out of control structures . • 
are not: allowed." • ... / ' |- .v 

Typically this second input -item is not directly supplied to a structure 
analyzer, but is incorporated directly in the tool's construction. -Therefore. «• 
substantial inflexibility is common. -•• \ ' " " 

4.9.4. Information output.. rError reports Tand a program call * .graph : are th£ * ' 
■', most common output items, of. - a structure analyzer. Error reports indicat*'-' 

violations of the standards that were input to the^tbbl.-- Calivgraphs indicate 
• the structure .of the graph with respect :ta : ^e'usle' ;: o^"a^ogiw; associated 
/with each -subprogram is information indicating all routines which call the 
•• subprogram and all - routines which are called by it. The presence of cycles in 

• the, graph (A calls B. calls ^ 

...never called are evident, as well/ as attempts to call nonexistent routines. 

In checking adherence to control flow standards, the structure -.analyzer may 

- also output a flow graph for each program, unit. The flow graph represents the 
, 'Structure of the program .with each:<:ont^dl- 'path 'in .the program represented by 

. • ,;an- r e^ge in -the graph'.-. Additionally, structurally , "dead" code 'within each r 
. module is detectable. •' ..4^>.-; v "; >; \ , .• c 

The flow graph and the call graph are items required as input : by data flow ' 
.analyzers, and it is. common for the "two analysis^pabmties&o be combined 
in a single autcmated tool. ■ . - . 
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,A»9.^* .Outline c€ wtbaQS ^pxk structure>analysis is * ah • automated ' static* 
analysis technique, little user action is required, aside f ran providing the 
input information, the user is pnl^required to peruse the output reports, and 
*y determine if progrm changes aiaMMred. "Some simple manif estatidns of -the 
tool.. may. not .fexprovide - detai^iPanalysis?^ reports;^ 
responsibility placemen the user to examine^ for exa^ the call graph 
for^the presence of cycles. : * • *V ™ 

V- :•" ': - T -r ; -: ." V'TvV- - • v;- ' V.'. • 

4 v 4.9.6. Example. ? . r •" v *. ^ .. . 4 \ ./'• 

i; ai j M online management information system program, figure 4.9^6-1,, 

'. calls a 'routine MAX -to report the largest stock transaction of the day for a 
given issue. if ..MAX does" not have the -necessary information already 
• available, .RINPUT is ; called to read the required data. Since RINPUT reads 

. many 'transactions for many issues, a sort routine is utilized to aid inl 
organizing the information before returning it -to the calling routine. 3 Due to -\ 
a keypunch error the sort routine calls routine MAX (instead of the proper 
routine. MAXE) to aid in the sorting process. -This error will show up as a 
cycle in the call graph and will' be reported through use of a structure 

, analyzer. -%i .. >: V : r ••' 



i ; mis ir; , 



!< _ J 

— — - ! 



! 'MAX- ! ,,-,;'! :•> •■ ■ 



! 
! 

1 v 



! RINPUT! ! 

Z ; r- ■ I 

' .'•- ..! . :. , !'. 



! MAXI ! \ ! SORT ! ! 

' ^ - — ! 

. .•/•••\---.>;' !— — ^-->! 
Figure 4.9.6-1 MIS Flow .Chart* 



b. As part of the pr<^araning standards formulated for a project, the 
following, rule is adopted: ^ ¥ - " : * ; > 

"All jumps from within a "control structure must be to locations 
after the end of the structure." * ' ■■€'" ■, y . 

Figure,'4.9>6^2, : a segment of, Pascal code, contains a violation of this rule 
whicti would be reported by a suitably constructed structure analyzer. 



■ v., - . • : 



5.2-. • i 



if. 2 ■;« 5 ihfin gOte TOO;. 



Figuire 4.9.6-2 Goto Violation / '•. : .: ■■& r -.--- •■'""-^x ' - 

*;9; 7 rj E^ectiveness. i : The taxtoque i^pnpletely reliable for detecting 
violations - of the standards _specified a^liiputi lhe standards, however, only 
cover a snail range of programming standai^ita^^ssible error situations. 
Ihus »l,,the technique is useful only in verifying very coarse program 
properties. The technique's prime utility,, therefore, is in' the early stages 
of debugging a design or code specification. - . ' , ■ : 

4.9.8. Applicability. The technique V is generally applicable and may ,be 
a^l^in design and coding phases. Particular;appliciability is indicated in' 
systa® involving large numbers, of subprograms and/or complex program control 

J* 9 -?' - teaming. "Minimal tracing: is required for. use of the technique. See 
■ "Outline of Method." •• ■" •'-.«;-... , 1 , : , "- j •, 

4.9.10. Cost; Little human cost is involved as there is' no significant time ; 
spent in preparing the input or interpreting the output, --For an average 
program computer resources are snail ;the processing rrequiredVcan- be done 
very efficiently- and only a single run is required for analysis. For large or" 
complex programs, the cost can be quite high. A plotter, which produces the" 
most readable stnicture diagrams, drives the cost up* 

4.9.11. References.' .'. . " : . 

c ; (1) FAIRLET, Richard E.,* "Tutorial: Static Analysis and Dynamic 
Testing of Computer Software, » £amaatfir>ol. 1 1', Now 4, pp. 1%23, April, 

.. „ • s .. . ■ . ...... > * -■<■. 

(2) HCHDEN,W.E. , "Reliability of the Path* Analysis 'Testing , Strategy", 
IEEE Transact ong £D Software Engineering, vol . SE-2,no. 3, 1976. 
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& U.4Q.$.. y Same: Cross^tference^G^erators ' ' .-."v. -y'i-Z '•• 

V.^-'-£;: ■<• : .. ■ £*" ';. ''• " • . :' : • 

k^: 2 " ^^ C ,*W*'* iS - - ! > os ^^rence generators" proauce,"'i^ibf data " 
n^^d^abelis showong an- o in* a program. . 

^^^T^^^ 11 input.' jnput to.cros»^^«nbe generators consis te of a * 
computer program iji jgither sourcse or; object format. P - ;- : 

il^Sv-^ 6 ^^ 0 " ;0 ^ f,ut - frdn a crossfreference 'geWator. r -is ah x 

X?* U ? of var iable names, . procedure names, and ..statement labels 7 "" 

^n^S° n lV e program- where they are defi^^and^- referenced. " 
'^iiSSn D '^ include,- xs da'ta type, attributes, and. 

: JJS^^S^"^ M ??° d 't Cross-reference *ene£l»rs;; provide • . useful : ' 
onf^taon which can aid both program. development and maintenance. -Ihey aid- 
program development by helping identify.errors ; such as misspelled identifiers • 

.im^Perly ^typed. .variables. Prdgram.maintenance is aided by helpSg to 
Jocate,. by variable or statement. label, those portions which "may be -affelted ■ 
yy aerogram change «*e.g.,,-a variable name; heeds to be changed) . V -;f 

. Cros^reference.generatbrs^e "widely avanable and are usually provided- with'' 
fTo^a^yzere e? Xt analyzers su< * as. compilers, standards checkers and data V 

Cross-reTerence listings should , be phecked in detail- after . a program change > 
has been.made > to ^check for misspelled identifiers and [ incorrect usage, etc? % 

4.10&6V Example. > " . ; • V 



-4. APPiicatibh. . A -communication network .controller manages the 

™° "5° lyLfi ne ^ ork . of high-spe^ccmimjnica a large' 

number of CRT terminals to an airline^eservatibn. system computer. f ; 

;'• b * i> .% or V :.* A variable used to'store message addresses is a^gned • ah 
address which- erroneously points to a location storing highly critical queue 
- ^trol-fformation. A subsequent call tp the device 'ha^dleTcauses ^ati to 
-. be read into the -critical storage area causing a system crash. I ■ ' , 

• J. : • c *^; Erh5r discovery. A" quick study, of' software's -cross-reference 
t t nK 1 snc * ,ed f 11 the locations where the offending variable was used, onfe of 
which clearly showed- that the. error -was due /to improper^ use of " a Writer 
variables, • . / • - ■- '. • . 

Figure 4.10.6-1 shows* -a sample program., listing and corresponding 
■1252? - ?t • 1316 P^cgram. is -a utility routine used by ilarge 
p^ y ^V^S yS1 L P ^^' ^ e ^^^ is -called 

P ™™.J> 2 l.^^&rt°™^ analyses. Ihe list shows. for 

?f^^ d ? ito t^ (e,g., integer ordeal), "usage • (e.g. , variable or ' 

function), attributes, (e.g. , .^argument, • whether the. variable has beeri set, . 
scalar or array) and, toe line nunbers where it is referenced. . \; 
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PFOBT VERIFIER '3/ 1 5/75^ERSION ,^ 
> \** ~ G DRIVER PROGRAM TQ TEST .EUCLIDEAN NORM FUNCTION 



1 



. ,3 ;. ' ••' 
" 4, T 

5 10 

c • 

6 ... 

7* 
8 

•• '9 
10 .-• 
ii 

12 20 

M4 2 
'15 - ~ 30 
■V 16 > 

t7 



INTEGER X( 100) 
r LOGICAL ERR 



' COMMON/ERROR/ERR - ' 
READ(5Y10) I 
F0RMAT(I2,I5) , 
< END, OF DATA CHECK 

'iFCi.Gr.ibo) stop* 

READ(5,10) (XCJ) , J=1 ,1) -* 
ERR=.FALSE: ' 
ANS=ENORM(Ij X) *?' ( 
IF (.NOT. ERR) GOTO 2 ' 
WRITE (6,20) • >-• • • 
FORMAT (15H BAD VALUE OF N) 
'GOTO 1 

: WRITE (6,30) ANS 
FORMAT (6H N0RM=,E15.7) 
GOTO 1 
END . 




PROGRAM' UNIT »MAIN 



NAME TTPF 

ANS R 

ENORM R 

ERR, EL 

I I 

J I 

X EI 
10 

V 

20 . 

2 

30 

COMMON BLOCKS 
ERROR ERR 



USE 




ATTRIBUTES 

SS ' 



SS 
SS 
SS 



\ ■ 

REFERENCES 
' 14 



9 

2 *3 



SA1 



.4' ;.:'6 
1 >7 9 

4 r 5 7 

4 . 13 16 

11 12 

10 14 

14 15 



**** » 
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Figure 4.10.6-1 Sample "Cross-Reference Examples 
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column 1; ' 



tela Figure iijj£L£=l 



Use Key r 
columns 1, 2: 



E explicitly typed 
cblunn 2: ' 



FA 
FN 







E ' 


-I'-'" 


INTEGER - 




R 


REAL 


GT 


D 


DOUBLE PRECISION 


IF 


C 


COMPLEX 


: ' . SF 


' V : 


LOGICAL . 


: •• • SN 




' HOLLERITH 





Attribute Key 



column '4: .. *" " 

C in COMMON 

i '• . . * ..." *■ 

colurm 2: ■ r 

E in an EQUIVALENCE statement 

coliaan 3; 

A dumy arguinent y ■ 

colurm 4: * ; 

S value set by program -libit 

column 5, 6 : 

S ■ ^alar / '~ \ 

An array with n <Iimensioris ... 



arithmeti^stetement . * 

function argument 
function name 

external (function or subroutine) 

'""■*-, • «' * 

assigned goto variable -V 
intrinsic .function 
arithmetic statement function 
subroutine name 

variable w ; •' ■•-•*■ - v 



w- ... * 
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Figure 4.10.6-1 Sample Cross-Reference Examples CCpntinued) 



4«10«7« v Effectiveness. Cross-reference generators 'are most '^ective- during 
the software maintenance phase to help determine where soltware errors are 
occurring, as seen in the previous example. Cross-reference generators are 
tools whose utility can often be . taken for granted or even considered 
bothersome (e*g. , "it produces too much paper") • Its lack of availability, 
however, will painfully demonstrate how^ necessary ^ this seemingly- basic ' 
capabili^gr is. Nwertheless, its true effectiveness is ; totally det^ende^bipon 
the quality of its use. 

4.10.8. Applicability. This method is generally applicable. 



: - . / . Page 49 

4. 10^9. 'Learning. Minimal effort is required to learn hew to *■ effectively 
utilize cross reference generators* 

£ 4. 10.10.* Costs. Cross-reference programs are widely available, usually as a 
t'" function provided by a larger system (e.g., a compiler) and add only an 
- incremental amount to the total cost. 

- *~ 4.10.11. References. • 1 

(1) RYDER, B.C. and HALL A.D., "The PFORT Verifier," Computing 
, £ci£D£& Technical Report, Mn. l2,Bell Labs, March, 1975. 
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4.11.T. Name. Data Flow : 

4.11.2. ^Basic features. Jteta flow analyzers are tools which) can determine 
the presence or absence of data flow errors; that is/ errors that are 
represented as pjarticiflar^> sequences of events in a program's execution. The 
following description is limited to sequential analyzers although efforts are 

^-^-under^iray^t^ — 7* — 

4. 1 1 .3. Information input. Data flow analysis ; algorithms operate on 
: "r:- annotated graph stra^ represent the program events and the order in 

v which they can occur. Specifically, two types of - graph structures- are 
. : required: a.v set of annotated flowgraphs and a pro-am "invocation (or call) 
graph. There must be one flowgraph for each procedure. *" >£ flowgraph is a 
digraph whose nodes represent the execution units (usually statements) of the 
: procedures, and whose edges are used to indicate the progression of -/execution 
' units. Each node r is annotated with.; indications of which progrM events • 
occurred as a consequence of its execution; The program ^Lhvocatioh (call) 
graph* is also digraph^ v*iose purpose is to indicate which procedures can 
invoke which others; Ite/npdes represent the procedures of the ^prpgram and 
, ite edges^ representee invocation relation. r -■. : h . 

4.11.4. . Information output. The output of data flpu analysis is ~a repprt on 
the presence of any ; specified event sequences in the program. If any Such 

- sequences are present, then the identity of each sequence is specified and a 
sample * path along which the illegal* sequence can occur is used.. Thejabsence 
of any diagnostic message concerning the presence of a particulaf^ event 
sequence is a reliable indicator of the; absence .of that sequence. 

4.11.5. ' flow: analyzers rely basical^ upon 
. algorithms from j prc^am^o^^ apy two particular 

:f sp^if ied everite can oc flowgraph annotated 

with all events of interest, these algorithms focus Upon two events and 
determine;" 1) whether- there exists some progriam path along which the two. 

>v, occur iii . sequence, and 2) whether on all program paths the two must pccur in 
sequence. : 3f one wishes_to determine illegal event sequences of length three 
: ; : or more, these basic 7 al^rithms cob ;beJapp:U succession. W 

A major difficulty arises in the having more than one 

procedure, because ; the procedure flbw^aphs often cannot be completely 
; annotated prior to data flow analysis. ELowgraph nodes representing procedure 
irivbcations must be left either partially or completely unannotated - until • the 
flowgraphs of the procedures which : they represent have been analyzed. Hence, 
the order of analysis of the program's procedures is critical. . This 'order is 
determined by a posl^d^^ in which the 

. bottom level procedures a^ 
so forth until the main level p^ 

date /flow analysis algorithms 1 must detera^ possibly - 

occur both first and last and then make for , 

.annotation of- .all nodes representing invocations proc^iire. Only in 

this way can i£ be assured that any possible illegal event sequence ~ will, be 
• . - determined. * . "•• . / 
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^il^^^tt I? e example, of - the, application of data flow 

In^hS ^^S 6 ^covery of references ^uninitialized program variable™ 
S /ase, the P r °gra^ events of interest are the definitionof a variable 
the reference, .to a variable, and the emission of a defStioS of a Jariable' 
Hence all procedure flowgraphs ; are annotated to iindSS whic? St ' 

^.•: erent .^eonthm is also used to determine „ if a specific variable 
definition omission must, along all paths, be followed * by" reference Sto^ 
?jT^« g For -invoSd procedures?^ aSSS ar^lso 

.used to identify which parameters and global variables ^are scmetimeTuSri 
always used , as- inputs and outputs, Into- information I s useTS^SSSe 
S KS^& *> enable Sil lf . . 



TV, 5 ' ' 

2?%8? ^"fyps might also be applied to the detection of illegal sequences * 
of . operations in programs written in languages such as COBOL Here the 

operations of interest would be opening, closing defining (i.e wrtofnS 
;:andjr^encing Ji. e reading) a file, ^rors wn^Ves^ or^Sence 

v eimned W , 0uld include ' attempting to use an unopened mf 
attempt to use a closed file, and reading an empty file. un °P eneo .^|iJ.e, 

4.11.7.. Effectiveness. As noted, this technique is' capable of determining 
the absence of event sequence errors from a program, or their presScTS f 
program. -When, an event sequence error is detected, it is ^Twfyl ^etec?ed 
SSabSL^oaths Because^ toese techniques 'do WitE'lS 

h^ncf ^i^rSe^' the ^ or ^ be detected on an ^executable, path and " 
hence give rise to a spurious ( message. Another difficulty is that this 
technique is unreliable in distinguishing individual elements^ of ah am£! 
Hence, arrays are usually treated as if they were simple variables As% 
SS e ' illegal sequences of operations onspecific^y elemSte maf bf-. 



4.41 8. Appli^lity. fData flow analyzers can be applied to any annotated 
K^i^r^ 6 ' ^ ^ailability , of this technique toon^limto^aS 
nSS^f * $*J» annuity of the (considerable) Stools and JechSques 
needed to construct such flowgraphs and call graphs. w*nniques 

4 ll T *^ tearningv : jlhis technique requires only a familiarity with and ' 
understanding of the. output messages. No inputdata or user Ser^Son to 

4.11.10. Costs. This technique requires computer time, " but the algorithms 
***** eff f cient ' generally executing in'time whSh irf^ea^ 
tZTr, * ™ 1 *° P^am size. Experience, has., shown, that the construction of 
the necessary graphs can be A considerable cost. factor, however Potential 
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As noted above, 110 hUBan Input or interaction is required,' resulting in only 
the relatively low human cost. for interpretation ^ . - • - 

4; 11.11. References. * f ':>': 

'■' -.-•i',r:i •"; : f ■ ■ ,,. . 

•<D OSTERWEIL, L.J. and POSDICK, L.D., "DAVE - A Validation. Error ' 
Detection,; and Documentation System for Fortran Programs. " Software. Practice 
"TifrElPJSri^fe T976. ■- * ■ -™ wmi ^. rrasree 

• : ■ ■:< (2) FOSDICK, L.D., and OSTERWEIL, , : . L.J. , .flDajta Flow Analysis in 
Software Reliability^i£Jl £fflBia£iBg Syorgya^, ^. 305^330, September 1976. 

(3) HUANG^ J.e., "Detection of Data Fl|w Anomaly Through Program Y 
^trumentation", iEEE Transactions SsfJaaCfi Engineering ,™. SE-SyNo. . .3, 
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4.12.1. Name. Execution Time Estimators/Anaiylers*;. ^9 

%A2.Z. : Basic features. Execution time estimators/analyzers are tools which 
provide ^information about ' the execution characteristics of a program.?- They 
carr be considered as validation tools in that they' can beV used to validate 
perf^majce • :; requirements and are part of' the programming' phase of the \ 

4.12.3. Information input. .The programs which are to have their execution 
performance monitored are, essentially, the input deeded by-", the tool. 
Depending on the sophistication of : the particular tool being used, the 

• programs may be processed by a processor which automatically inserts probes to 
measure performance or pnobes may- need to be manually inserted.* The probes 

■■ usa ^ consist of calls to a. monitor which records execution information such 
as CPU and I/O -time, and statement execution" - counts, '"'h . 

4.12.4. Information outputs. The output produced byV s execution , time 
estimators/analyzers are reports which show either by statement and/or module 
the execution timejdistribution characteristics; ■■ For example, a tool will 
provide .information showing 'per module the number of. entries to the; module, 
cumulative execution time, mean execution, time per -entry and the 'percent " 
execution time o£ the module with respect to the total program execution time. 

4.12.5. ; Outline of mett6d.j;:Executiori time- estimators and^ -execution time' ?" 
analyzers both perf orm\ similar . functions but in different" ways. Execution 
time estimators (1) function much in the same way as. test coverage analyzers 

source program is instrumented with .probes which collect statement execution 
counts when executed. Associated with, each statement is a < machine dependent 
estimate _ of the time required to execute the. statement. The execution time 
estimate is multiplied by the statement execution count to give an estimate of 
the total 14me spent executing the statement. This is done for all statements - 
in a- program. Reports showing execution time breakdowns by statements module, 
statement type, etc. can be produced. ~ : : : • ' 

Execution time analyzers are not usually as sophisticated as execution time 
estimators. Probes to measure the actual execution: time of modules or program 
segments are inserted (usually by hand) into the source program, • When the 
program has ; completed its execution, but just before it terminates, a routine v 
is called which prints a report showing the execution characteristics of the 
monitored portions of the program. , ~~ 

The value of tiie. .tool 1 ie& primarily in its use as a performance requirements * 
validation tool. In order to be used to formally validate performance 
requirements, however, it is necessary for the performance requirements to 
have, been clearly stated and associated with specific functional requirements. 
Moreover, the system should have ;been designed so that the functional 
requirements can be traced to. specific system modules. 

•Assuming that-the above conditions are met, the tool could be used* in" the 
following way. , The program to be analyzed would be monitored by the execution 
time estimator/analyzer during, testing. The execution times for the modules 
corresponding to specific functional requirements would, be compared with the: • 
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perf crniai^ requirsnent £or T . that function. .Ihos^iymoi^es which fail - to 
satisfy their performance^ r^ detail for 

possible efficiency improvements.. , -The tool results can also help to identify 
execution time critical sections of code. Once the necessary optimitions have 
been made, the program, should be again tested using^ the tool to validate the 
performance requirements. "'. - A >-?:-'-r - ? 

4,12.6. Example. — '— * , "'■ — — ^ X '- : J V. 

a. ] Application. A particular module in a real time, .'embedded 
computer system is required to perform its function within a specific time 
period. ....If not,., a critical time dependent activity ^cannot be performed, 
; resulting. ^ the loss of the entire system. > : ^> 

Err&r.: The module in question contained an error " iwhich involved 
performing' unnecessary ■ ccn?>ari5ons^ . during a table look-up function although ' 
the proper table entry was alw^s f ound^- , ; w - ^ v ^V;^ 

* v c.; Error detection. The problem was discovered^ during system testing • 
using an execution time analyzer which clearly indicated that the offending 
module was not able to" meet its performance .requirements. The specific 'error 
was discovered on further examination of tifte module. ' s • 

.^t^7^- Effectiveness. The use of execution . time estirators/analyzers (as * 
WeElvds test ^coverage -analyzers) has ^cwered an interesting property of many \ 
programs. The majority , of the execution time spent ly>-a program is r spent* 
executing a very small percentage of the code. Knowledge gained of where this 
execution time critical code is located thrbtfgh the use of an execution time 
estimator/analyzer can 6e extremely hripful in optimizing a -program in order 
to satisfy performance requirements and/or reduce costs. - 

4V32.8. Applicability. Execution time estimators/analyzers can be used in 
any application. 

4.12.9. Learning. The learning required is simply that which* is necessary to 
execute the tool. 

4.12.10. Costs. The tool is automated and therefore does involve seme cost. 
.The amount will depend on the tool's sophistication, but generally will not be 
excessive. * " ■ 

4.12.11. References. 

(1) "PPE Users Guide", Boole and Babbage, No. U-D503-0... 

v (2) "Poseidon MK 88 Fire Control System Computer Program- Verification 
and Validation Techniques Study", Vol. Ill, Ultrasystems, Inc., 500 Newport 
Center Dr., 'Newport Beach, CA, Nov. 1973. 
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#*i3.1. Name. Formal Reviews. 



4*13.2. ; Basic features. - Formal reviews constitute a series of reviews of a 
; syst^, .usually conducted ; at major milestones ins the system development 
JJiecycie'. , They are used to improve developments visibility and product 
quality and provide the .basic means of communication- between the project team 
COTpany managanent, and user representatives, They must provide judgmental 
_decisi«jSrfflade^aHasam^ 
current system operations. Formal reviews are most often implemented for 
medium- to large size development projects/: although small projects often 
employ a less rigorous form of the technique. • 



The. most common types of formal -reviews are held at the completion of the 

T^^^" te ' u^^^ 7 ^S 0 ' detailed (Critical) Design, Coding, and 
Installation phases'. Whereas names of these reviews may vary by company? some 
generally recognized names are:. Requirements Revfew,* Preliminary Design 
Review (PDR), Critical Design Review (CDR)-, Code Construction Review, and 
Acceptance Test Review. . , . t, j?k 

4.13.3. Information input..' ;Sie input- to a particular formal review will varV 
slightly depending on the stage of the lifecycle just completed. In general/ 
eadh formal review will require teat some sort of review- package be assembled 
and then distributed at a review meeting. This package commonly contains a 
summary of the requirements which are the basis ' for the product being 
reviewed. These and other common inputs to-formal reviews fall into :three 
main categories, described below. - • '.. ^ .. "• 

a.. Project documents. These are documents produced by 
development team to describe the system. The specific documents required, 
dependent upon the lifecycle -phase just completed. For example: • a <ev 
conducted at the conclusion of the requirements phase would necessity 
availability of Functional Specifications, or System/Subsystem Speci fications 




b. Backup documentation. This type of input is documentation whi 
is not usually contractually required, yet preparation of which is necessar 
to support systems development or otherwise record project progress. %gecifi 
types of backup- documentation vary by the phase for which the review . i^ 
performed. Rough drafts of user, and maintenance manuals are exampl^t of ^^ 
backup documentation examined during a design review to plan for continuation *^ % 
of toe project. Program listings are an .example of backup documenteM>n ' *f V 
utilized during a code construction review. ; j£ * ;•. J. 

c. ; Other inputs. All'other inputs are primarily used to clarify* or 
expand upon the project documents and backup, documents. They. may include - 

rviewf oils and slides, prepared by; project management for - the formal review* 
meeting, ^toe ; minutes of the previous -phase review meeting, or preliminary^, 
■evaluations of -the~ project documents under review. • 



4.13.4. Information output. The information. output associated with a formal' 
review generally falls into the f olJfawing categories. . 
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* a* .Management reports. These are written reports frcm the project 
manager to upper management describing the results of the review/ problems 
revealed, proposed solutions, and any upper management 

V b. Outside reviewer ^reports. These are written reports to the i 
project manager from participants of the review who have not worked on the 
project. T&ese reports provide outside reviewers an opportunity to egress 
^eir^appraisal^o 

objectives. It also allows them to make * suggestions -for correcting any 
deficiencies noted. ~ ' * 

i\ ; c. Action items. This is a list of all required posWeview action 
items to be completed before a review can be satisfactorily closed out. With 
each item is an indication of whether customer or contractor action is 
required for resolution. 

d. Review minutes. This: is a Written record of the review; mating 
proceedings which are recorded by a designee of the leader of the review team. 
The minutes of the review are distributed to each review team member after the 
completion of the review meeting. A ; . ^ * 

e. Decision to authorize next phase. A decision must, be reached at 



- any formal review to authorize initiation of the next lifecycle phase* 



f. Understanding of project status.. At the conclusion of any formal 
review there should b£*;a~ cannon understanding of project status among the 
.project personnel present ^at -the review. ^ 

4. 13.5- d^ine of methba^ v ^ > 

a. Participants. The participants in "a formal jreview- are often 
selected from the following^ gr^fip bf peopler " ^ 

o Project manager ' " : %d 

o Project technical lead . ';■,£>& . ?v' 

o Other project team members - analysts, designers 
o Client 

o User.Vepresentative(s) 
p ifene management of project manager' 
o Outride reviewers - quality assurance personnel, ej 
^ other projects ' ^ - 

*o Functional support personnel - finance, technology 
o Subcontractor management, if applicable k 
o Others configuration management representative, *i 
representative 

:^b. The process. Formal -reviews should be -sche 
project management. Each review must be schedule 
dxxrlxig The review effect! vely 4 

milestc^-for any Articular phase. '-tf«v/ 
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Ibere are five basic steps Involved in «w»ry fo rmal " - - 

1 . Preparation. All documentation that serves'as source material for 
the^ review must, be prepared /prior to the meeting..- These materials may be 
distributed to each « participant tefore the meeting in order to allow 
sufficient- time to review . arid make appraisals of the materials. The location k . „ 
and time of the meeting must be established, participants must be identified : * 
_andLan_agenda planned. - : - : -. : -. . . . ; — 

•S 2. Overview presentation. At the review meeting, all applicable 
Product arid Backup Documentation , is distributed -and a high-level /sttonary of 
the product is presented. Objectives' are also given. - 

^5-v.!^3.. DetaHed-presentatton. A detailed description of the . project ' 
status and progress achieved during the review period is presented.' Problems 
are Identified and openly discussed by the team members. 

':, *■ ' ' ■ ■ ' v - ".. ■(:■'.>■■■ . -•■ ; x> ■■■■■ 

. 4. Summary.". A summary of the results of \the review is given/ . A 
decision; about the' status, of the product is made and a list of new action 
Items is constracted and responsibility for completion of each item is 
assigned. - r t 



5. 'Follow-up. -The completion of : all action [ items" is - vearifiedV" - All " 
reports are completed and distributed. ~~C^f : V ; 

4.13.6. Example. By contract agreement, two*weeks prior to completion Of the 
requirements document, the producer of a program receives notification, frdin 
his client that a requirements review meeting is desired. The client notifies 
a preselected chairperson to conduct the meeting. For participants "the * 
chairperson has selected the project manager, project technical lead, a member 
of the requirements, definition team, and a member of the requirements analysis 
team. The client also has indicated that he would like - to .'include the 
following people in the review: a .representative from the user shop, a 
reviewer from an independent computing organization, and a representative from 
his own organization. . . 

The chairperson informs ail review participants of the date, time, and ^ 
location of the review. Ten days prior to the meeting, the chairperson 
distributes all documents produced by the requirement%definitiori and analysis 
teams (requirements dodumenV preliminary plans, other review material) to 
each participant. In preparation of the , meeting, each reviewer critically 
inspects the documents. The user representative . is puzzled over the inclusion 
of a requirement- concerning the use of -a proposed database. The reviewer from 
the outside computing organization notes that the version of the -operating 
system to be used in developing the system is very outdated. *A representative 
of . the client organization has . a question concerning the use of a' 
subcontractor in one phase of the project. Each reviewer submits his comments 
to the chairperson before the scheduled review meeting. The chairperson 
receives the comments and directs each to the appropriate requirements team 
member to allow proper time for responses to be prepared. . 
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The requirements review meeting begins with a brief introduction by the 
chairperson. _ All jrarticipants are introduced, review materials are. listed, 
"and the procedure ^ A presentation is . 

then given suno^izii^ the ' prbblem &at led- to the requirements ; and the , 
procedure that was used ttf fe^ r^uirements*. At. this tifee, tile -user 

representative inquires about the requirement concerning the use of a 
particul^data^ase as stated in the requirements documents' The project 
^technical^s^ 

this re&porise, which is so noted by the^ recorder in the official minutes of 
the meeting . : r ' 

The meeting continues with' an analysis of the requirements and a description 
of the contractor's planned Approach for . developing a solution to the problem, 
;;At ttiis^tQne, the questions from the client represe^tetiv^ and the outside 

computing organization are discussed. The project manager responds to 

.questions concerning the use of a subcontractor on the project. vCertain 
^suggestions have been made which require the approval of the subcontractor. 
iThese suggestions are placed on ^ the action list. The technical lead 
^ack^ledfees v the problems t that the .independent computing* organization has 
Vppinted out. He^notes .that certain system . vendors must be contacted to 

jC^pL^e the problem. - This item is also" placed on the action list. A general 
..di^ussion among all review team members follows. ' \ 

At the end of the review, the chairperson seeks a decisSfc^fran the reviewers 
about the acceptability of the requirements docund^^ They agree to give 
their approvial, providing that the suggestions noted on the action list; are 
thprou^ily investigated. . All> participants agree to this decision and the 
meeting is. adjourned. * ^ \ ; : ■ ■■ : : -- • * 

The chairperson distributes a copy of t^e minutes of the \meeting t including 
acjtidn items, ; to all participants . r The project manager informs jthe 
subcontractor of the suggestions made : at the meeting. ^ The subcontractor 
subsequently agrees with the suggestions.. The project technical pleader ' 
contacts the system vendor /from which the current operating systwn was 
purchased and learns that the latest version can be easily; installed before it 
is needed for this project. Be notifies thea project manager of this, who 
subsequently approves ~ m its I purchase; . The /requirements * document is 
appropriately revised to "reflect ; the c^pletibhv these action items. " The * 
chairperson "» verifies that all action items Jhave been completed. The project • ' 
manager submits a Management Report to .management, summarizing the review. 

4.13.7. Effectiveness; Since the cost to, correct an error increases rapidly 
as the development : process progresses, ^detection of errors by' the use of 
formal reviews is an attractive prospect. ^ \- m 

Stime ofrthe qi^itottve benefits attributable to the use of formal reviews are 
given bfelow: • ■ . -f ' } t "■" ' \-; ; 

b Highly visible systems development . ^ / 

. o Early detection of design and^^:^ errors 
jo More Triable estimating and scheduling ; 
o Incr^s^^prbduct reliability, maintainability ■"•^^.^'r ^ . 
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, o Increased education and experience of all iattvid^ in the 

■■■ process l^I'-C^* ~ \ ", / .v > ■ 

o Increased adheraace- to standards^ & 

*' o Ihcreased-user satisfaction .;.>■■ S • v •-, 

Little data is. available ufaicb identifies the quantitative benefits 
attributable to the use of formal i^iews/ - ♦ 

Experience with this technique indicates- that it islnost effective on large 
•' proj ects. The costs involved in performing formal reviews on small projects, 
however, may , be sufficiently large enough to consider" lessening the formality 
of the. reviews ,or ey.en eliminating or combining some of them. 

• p. -13. 8. Applicability . Formal .reviews are applicable to large or small 
projects following all development phases and are not limited by project type 
or complexity. • • : . , v ' : '/, • . ' . 

*.|3.9. Learning. This •technique does hot fc require V any special training. 
However, the success or failure of a formal review is dependent on the people' 
-who 1 attend. They must be intelligent, ; skilled, Jajowledgeable :, in' : a specific 
problem area*, and ..be °.able to interact effectively with other team members. 
The experience and expertise of the s individual responsible for directing the 
review is also critical to the success of the effort. 

■ .■ """■".**- ' " * ...'■».■ 

4.13.10. Costs. The method requires no special tools or equipment. The main 
cost involved: is that of human resources. If formal reviews are conducted in 
accordance "with the resource guidelines, expressed in most references, the cost 
of . -reviews for average programs are not high. However, the cost of reviewing 
major programs can be significant. • Most" referetwes suggest that formal review 
meetings should not require mor> than 1 to 2 hours. : * Preparation time can 
amount to as little as 1/2 hour and should not require longer than 1/2 day per 
review. . '. : •• ••' ■: 'si-. •:' 
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V r'4.14;2. :;Basic features, the purpose of -formal verification is to ' apply the 
;;v formality.- and rigor of mathematics to the task of proving the consisjtency 
'^"between an algorithmic solution and a rigorous,' complete specification of' the 

^tent of the solutton. ; CijJs;^ * ' 

are ^h e 

>- solution .specification" and thek .intent specification. The solution 
; • ^specification is in algorithmic form; often but not always, executable code, 
.'.■the... intent specification Vis descriptive in. form, invariably/ consisting of 
assertions, usually expressed in Predicate Calculus,^ 

• -Additional inputs, may be required -depending, upon 'the rigor Jjand specific 
' mechanisms' .to .be employed in the N consistency proofv For^xample, the 
semantics of the language used to , express the , solution specification are 
■ ' required : and m^^ degree of rigor consistent with the rigor 

V o£ tfre proof being attempted;-/ Similarly, simplification -rules 'and' rules of 
inference may be required as. input if the proof process is to be completely 

. , rigorous. • .- i: r V • .;. •.- ■ '•■ 

: v 4.1%4. Information output. The - proof cprocess may terminate with a 
successfully completed proof of consistency, or a demonstration -of 
ia»nsistency, or it may: terminate inconclusively, in^the former- ,twp cases, 
y.-- the jjropfs themselves" ' and . the ^prcye^?^ the 
latter case, any fragments chains: o^ -reasoning ''We the 

only meaningful output;-. v 7heir signiff^anee - aV^j^te^ highly variable. ' 

4.14.5. Otitline of method.. The usual ^^nietHtadr- /used." In .carrying out formal 
verification te^Floydls Method of Inductive Assertions or a variant thereof. 
This method entails the pa^tiqni^ Solution" specification into 

,/ algbriltacally r *traightline fragments by means of strategically placed 
.assertions. This partitioning reduces the proof of consistency to the proof 
bit. a set of smaller, generally nuch nK>re manageable W . 

ELcgrd f s Method dictates that the > intent of the, solution specification be- 
captured by two assertions. The first assertion is the input^ assertion which 
describes the assunptions about the input. The setfond -assertion is the output 
assertion, which describes. the transformation of the input, which is intended 
tq be the result of, the execution, of the specified solution. : 'In addition, 
ihl^noediate ^'as^tions must . be fashioned ^n^placed within the body of the 
solution specification ; in ajcb -a way ^^ ev^ '/ioop . in the solution, 
specification contains at Ueast onfe intermediate assertion. Each such' 
- intermediate assertion must express completely .the transformations which are 
intended to occur or are occurring at the? point . of placement . of the assertion. ; 

The purpose of* placing 4 the, # assertions 1 as ju$t . described is to assure that 
every possible program execution is decomposable into a sequence of 
straightline algorithmic specifications, each of which is bounded either 
end by * an assertion. If > it is known that eachv^terminating -assertion is 
^^ecess^rily in?>li^|>y executing the specif ied algorithm under the conditions: 
" tB * initial assertion, then, by induction, it can be shown that the entire 



fe«:,>.v , v , : ■••■•>• .... ■ 

1 J jexeoutioa behaves as specified, by the inpuVoutput' assertions, and hence as 
^intended". For .the user to be a^sured^^ this, Floyd' s Method directs that a 
' set of lennas be proven. This set consists of one lejnna for each *pa^r of 
-assertions \jhich is separated' by a str^ghttine algorithmic specification and ' 
- no ot h er inteCTening-assertion^For-r^^ pair; the lemma states - 

that, under the assumed conditions of the initial assertion, execution of the - 
a^orithm'^specifiedby the intervening code necessarily implied the conditions 
. Of £ the terminating assertion. -Proving all sue* lemmas establishes what is' 
known as "partiar.cdrrectnfew. f- Partial correctness establishes -that whenever " 
' the specified- solution process ^terminates, it has- behaved' as intended. In 
addition, total correctness is established by proving, that the specified 
* solution „ . process must always terminate. This is clearly an undecidable 
question; being equivalent to* the Halting Problem, and hencefits resolution is 
.invariably^ approachedjHjj^^ the application '■ of heuristics . 



Injttie above procedure, the pivotal capability is clearly the ability to prove-, 
the Various specified lemmas. This can be done to Varying degrees of .rigor, 
resulting in proofs . of , corresponding varied degrees of reliability^ and 
^tr^tworthiness. For ; the greatest degree of trustworthiness, solution • 
^specification, intojfc specification, and . rules of reasoning-, must alsT be 



specified with complete . rigor and Decision. The principal difficulty here, 
lies' in specifying the solution with^^mplete . rigor and precision. Ihis 



entails specifying the semantj^ \^ipie specif ication language, an<£ the *. 
functioning of any actual executions-environment-" with 'complete rigor and 
precision. Such complete details are* often difficult ;or impossible to adduce. 
Ihe^are, moreover, when available, , generally quite voluminous, thereby r 
occasioning the need to prove lemmas which are long and. -intricate*- . 

^M?r.6.' Example. As an example of what is entailed in., a. rigorous ^formal . . 
verification^ activity, consider the specification of a fbubble sort procedure.: 
(The details oT~this can be. found in Reference 3 for . this technique. ) The 
intent of the bubble sort must first be captured by. an ihjiut/output -assertion 
pair. Next, observing that the bubble solution a^o^ilhmicdntains two-.-soested' 
; 'loops, leads to the conclusion tha^ 

might- be fashioned, or perhaps one particularly well placed assertion might .." 

suffice. In % the former case, up . to- €d.ght ; lemmas would then need ;to>be 

established; one corresponding; to each of the (possible two)' paths' from the 
. initial to each intermediate assertion, one corresponding to. each^oflj^e two v 

paths from an intermediate assertion back to itself*: one for each . of the 
'. (fiossibly two) paths from one intermediate assertion teethe other, and finally 

o^ for each: of the (possibly tuo) paths' from., intermediate te terminating 

assertion. . 'Each -lemma would .hayev to" be. established through -rigorous 
. nfeihematical 'logic (see. Reference^ 3)^ ' Finally/ /a- proof. ^>pfE*^ecessary 

termination would -need toibe fashioned ^see Referoace 3) ; * 

y- 4.1^.7. Effectiveness. The ^effectiveness of formal ' : vefid^!^^^$^r been ;V. 
* ; attacked on several ' : grounds. •>'• First ' and most fun^na^^iy ^ 4 f^noal. :/ ' . 

verification can only, establish .consistency . between iltent^^aml^ solution •'. ■ 
, : specification.' Hence,- .^inconsistency ' can,' indicate error ih" r eithec- orbi&tii?; 
' The same can be^vsaid^or'most ^ otber verification^ Ixcita^v^^^^gir, -V^wKat''* k ' 
makes; .tEECs pal^cularly* daiiaging for formal verification iA^tat complete 
" rigor^jpd detail y ••ftag. .the: ^tent ^ specification ,-are* important; and; thifes. 
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requirement for great detail invites ,ereor,. • 

J e amount, of .detail also occasions the need for large, complex lannas, 
These, especially when proven using complex, detailed rules' of inference; 
produce-jrery large, , inteicatepr^f ; — -L 

Finally,; formal verification -pt actual programs "is* further cSnp^ated'by the 
necessity to express rigorously the execution behavior or ^e^t^^computink 
environment for the. program.'^- 'As a consequence of ■■ . . . *° 

environment: is generally; modeled incompletely and 
restricting the validity of Ihe proofs in - ways 
determinei •- 



'the 

sf^Ly^thereby 
^diff^ult to 

W4v" " ' 



A 



\ 



Despite these difficulties, a- correctly. .. proven set: 
consistency between a complete specification and a 
whose semantics are^ accurately known and- expressed' 
assurances *of correctness obtainable. This ideal- 
attainable by apply i»» . automated theorem * provers tb ' del 
rather than code. • \^ • 

4.14.5; Applicability. : Formal vesication" is a 
applied • to -determine- the. ; consistency between any 




iblishing , 
Ication 
ihe' greatest * 
Tee seems best 

whiph can 



specification, and any .- intent Specification. As / 'elaborated upon - elriier, 
however, the - ^stwqrthiness of- the '-.results is highly^yariablfe .defending 




4„T4.-9. Learning. ' vAs'iioted^ Sie essenqe, of tthis technique- is" 'mathematical . 
Thus, the gore nathematical 'sophistication and 'expertise: which^practitioners 
possess, . the better.^' Iw particular, a." considerable amorint^of mathematical 
training -and expertise is^ecessary for the results of applying this technique 
to be significantly- reliaae^d^i«s^rthy; " 



4.14.-10. Costs. Tnis- Jie*chniqu^;^toen seriously ^applied, jnust be expected to 
' co ^^ very ' significant amquritS' of \ the time and effort of highly trained 
matt^natically proficient personnel. -^tence^^considerab'le human-labor expense 
musj^fjbe expected* - • '- ' '• .<■?:- > -\ . 

As noted earlier, human ^effectiveness can jbe considerably improved through the 
use of automated tools such* as theorem provers. It is important to" observe, 
however, that s^ tools can -be prodigious consumers of computer resources. 
Hencej. their^ operational, costs.are also quite largp. 

4.14.11. 



AW ' 



Reference^ 
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4.15.1. Name; .Global Roundoff Analys 



Algebraic Processes 



4.15.2. Basic features. €Be technique involves the use of computer software 
to locate, numerical instabilities: ii| algorithms consisting of algebraic 
processes., .Global roundoff -analysis Is^he^determination-- of- -faow--rounding- 
errbr propagates in a given numerical method for nnny or all permissible sets 
of data. This technique -has two jt^as of application; Case I - to decide 
whether an algorithm is asacd&as can -be expected given the •'fundamental" 
limitation of finite precision^pmetic; and Case II - to decide which of 
.two competing algorithms MWsfib sta ble, " iVe. , less susceptible to rounding 
errors.. -^ZzaeiysSK..- *. ■ y-- 



4.15.3w Information inputs 




a. Case 1° — Analysis" Of. a single algorithm, 
Ci) algorithi^escribed a simple^paggranffi 
(ii) data,s€pfor algorithm - ; t^^^^:'^^ 
;. jCiii) choice' atid- type /of rounding ^Sw^flfesures 
Tiv) Stopping value f/>r jna^ tpfc^ig^? 



ig language 




b. &se H ^Cflmpapfsoh Of two algorii 

* (i) each algor itSai described in a .simple programming language 

(ii) data set' for algorithms^ ° ^ 

(iii) choice of rounding ernSr measure and-'mode of comparison 

(iv) stopping v^ue ^©fiynaximizei 

- ' • ■ v -•' ■"• ■ 

4.15. 4 . Information output. 



a. Case I - .Analysis of$a iisingll 

(i) output computed for the' initial data' set 

(ii) list of values found by the maximize^ " 

(iii) final set of data .*' r . 

(iv) if instability diagnosed, then all arithmetic operations at the 
final set \6f data are listed 





b. Case II ^Comparison of two algorithms 

(^X^tj^^pwted • for the initial algoritiia^^^pS 
(iiFfist of values .found "ty.the may imi ^^m^&^i^ 
(iiStbfinal set of data - : - 



For . an ,;|lgorithm and a data set,d r then: 



4.1515. Outline of method. 



(a ? A f ^ ction »* called 'a WilfcLhson number, has been defined which 
measures the effects of rounding errors. Large values f or w is the sign of an 
unstable algorithm. ,, . ■ " 

■ ' ■ 

(b) Wilkinson number has been shown to be a "smooth" function of cT, 
ri.e^'^as the original data set values are altered in 'anal] increments, the 
yalues>: of w are correspondingly altered in small increments. ..gi 



(c) ^ approximation to wYtfrinison ntnbere 6as been developed which is 
straight forward to canpute.' . . ....... H . .. .. -.-^ T~ 

(d) The representation of /the algorithm is analyzed. ^ 



. Ce) Using the iriitial data set as a starting point, the global 
analysis program uses numerical maximization .techniques to modify the data set 
The search is. directed toward finding a data set with a disastrously- - larae 
value of w(cD . • v . •;, * 

4.15.6.: Example. Triangular, Matrix Inversion (4). The better matrix 
inversion algoritte are .known f ran experience to almost invariably produce 

question remains whether 1 there' is . a 
guarantee^ that theriesults,are ^allffly^good. The question can be reformulated 
: as.: ,.%the troditional back substitution algorittm , for inverting, an upper 



fei ^^fepwmerjSaByi^ that 



bound, depending on-the ma^r^sSe^jSr^ w? : :^|c/: apply the test 
algorithm is reprejsenjedj^^y^ -prbgram in^figure 4.15.6-1. 
statement "TEST (N=4F^Sai|sgg^$^^ numerical^ 
be conducted in the domain Jpg|c43» be 
calculated. . .. - • 1 * 



C 
C 



TEST (N=4) 

compute s = ex mEss^^^^^^ib^mmmmppER 



TRIANGULAR MATRIX. * V- -1 ■ ^^m^m^^mW^^ 

,lJ),Ta^>) ;^^^^^^^^;:V. 



c 



DIMENSION (S(N 
INPUT T. 

FOR J = 1 to N BY'1 

FOR I = J to 1' BY -1 
INffiT (TCI, J)) 

END(J) 




•"0 



c 
c 



COMPUTE S. 
FOR K =1 TO N BY V 

S(K,K) = 1.0/T(K,K) - . ' 

FOR I = K-1 TO 1 BY -1 , . 

S(I,K) 0 = -SUMMATI0N(Ta,J)«S(J,K),J=I+1 TO K)/T(I,I) 
END (I) , ' _ 

END(K) ' 

OUTPUT JSi * 
FOR Jr1 TO N BY 1 

FDR I=J TO 1 BY -i 
OUTPUT(S(I,J)) 
••< END(I) 
,END(J)# 
STORvl 




Figure 4.15.6-1 Triangular Matrix Inversion 
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lilt compiler portion of the pacteg^lKecks. the" program for errors, then 
, JU - them into a. form suitable for analysis. 



Ihe initial data set for the search for numerical instability was: 

j A 0 0 Ov ' ■ ' ' ' . ' 

-^^•-^/---••-2 o ■ <M 



• [r^."* ■"'*; > ^' ''^'Jf"?"^-'- 



The roundoff analysis program was told to seek a valUfe of tf in excess of 
10,000. The 'nwYlmlTier located the following matrix:- :: > 

^ - y -0.001 * 5.096 5.101 - 1.S53> 

T<6 « • / 3.737 3*740 

0.0006 .15.254*/ ; * 



"5 



c 




with W, (TL)>10,000 in 6 seconds CPU time on a^&M 370/168.. \* 

The fact that Wi; can be Jarge for, data like T seems implicit in known results, 
' «.*•» (6), verifying the 111 behavior of triangular matrices with diagonal 
; entries approaching zero. £ 

4.15*7. Effectiveness. Failure ^ the -maximizer for find large values of w 
does not guarantee that none exist (2). Thus, the technique %ends to be 
optimistic; unstable methods may appear -stable. However, -experience 
indicates tiSt this method ^Surprisingly reliable. At least, the, failure of 
the maximizer to find large values/ of w «an^ be -interpreted W providing 
evidence for stability equivalent to a large amount of practical experience '"' 
With low order matrices. -v. \ : ■ 

4.15.8. Applicability. The technique is intended for noniterative methods 
from numerical linear algebra.^ • '*• 

4.15.9. Learning. Most algorithms should be able to be analyzed in 2 to 8 
hours of training and preparation assuming the software is available. 



'"4.15.10. Costs. The performance! of the technique is related to the 
-performance of the algorithm .being checked. • „' ':*•••, 
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. Mane. Inspection . . 

4JI6.2. Basic^features. Informal reviews constitute a thorough inspection 
mecha^sm usecT to detect errors in system components anj documentation. 

SS? ^^2S?25- aeli>S f cc T? ll y recognized inspections t ar7coSucted 
•Jur^ng the design and programming ^stages and are referred to as design 

SS^fo ■•-°3* S^^SP 8 -' However, the inspection concept may be 
^ p1 ^ to^functionally-c^ete part of a system during any or all ptoses 
oftae Recycle; and . are typified by utilization of checklists and summary 
. reports. Another unique feature of an inspection is the use of data^frcm oast 
' inspections to stimulate future detection of categories cf errors! ^ 

J'lf: 3 - Information input. The input required for each inspection falls into 
SS^tfSE*? Project dpc^/ ba^ 

. , a \ Project <toeume^g-These are - documents produced by ' the 
development, team' to descabet^e system. The specific documents' required are 
dependent upon the JLif ecycle^^currently in progress. For ex^mple?^ 
inspection conducted during the design phase would necessitate availability of 
. Functional Specif ications or System/Subsystem Specifications. ^ 

*b. Bgckup documentation. This type of input is doc'umentetion which 
is notusuafiy contractually required, yet preparation of which is necessary 
. Co support systems development or otherwise record project progress, " specific 
tges, .of backup ^documentation vary by the phase, in, which the inspection is 
conducted. Date dictionaries and cross-reference tables are examples of 
Backup documentation utilized during a design inspection. Program listings 
are^an example of code inspection backup documentation. ^\ 

- c. Checklists. Each member of the-inspection ^m uses a chlcKLi^t 

* w ^i* 1 " Preparation /and during -the course of the- inspection itself . The 
'Checklist content may^ vary .based upon ~ the particular* application , being 
inspected and is updated fran - feedback of other -recent inspections. For 
example, a. checklist^*) be employed during a code ' inspection 'of a COBOL 
program component would contain items like: ♦ 

-o Are specialized printer controls used to enhance component readability 
(e.g., use of EJECT or SKIP commands)? . - T* " 

o Does each procedure have only cto^exit and entry? * 
o Are IF-THEN-ELSE statements indented in a logical fashion?;: 
o Are file, record and data names representative of the information they 

contain and do toey conform to. estatt naming conventions? 
o Are comments expl®st~ and accurate? 



4.16.4. Information output. The information output associated with- an 

inspection is either related to inspection planning' and scheduling or 
inspection results. v . . - •. 

I n spec£ion--s(Aedifl^ 



from management that an inspection shbuld be forthcoming. -The memo defines 
the roles and respbnsibiliti^^ctf each inspection 1 team member-, estimated^iiipje 
required for each inspection task^ and- a sunmary of the status of the item 
being reviewed (including any previous inspections: conducted). 

b. Problem Definition Sheet/Error Descrip^ This f^nn is 
used to record information* about each detected error. r£Et describes the 
location, nature, and classification of the errors. T^- ' : - 

c. Suanary Report. A Summary Report is used to .docunent- -correction 
of all errors reported during an inspection. Data recorded oh the report is 
tabulated and becomes part of emulative error statistics which can be used to 
improve the development and: inspection prcspe^sesi - 



d. Management^ Reports. Tfie^ : ^r^^ts are -the means' by 
management is informea^feout the types d^^ors being detected and the 
of resources being expended to correct them. ^ Ihe information from these 
reports highlights freqtaent sources of- errors, providing input to management 
for future updates to the inspection checklist. 

4.16.5. Outline of method. , . , ^ . . 

* * - • 

L a. Roles and Responsibilities. The group, of people responsible'* for 
the inspection results are usually called an inspection team and are given 
responsibilities based upon their contribution to the item bfcing inspected .\ 
The leader of the group is responsible for all. process planning, moderating, 
reporting, and follow-up activities. :. The designer /implementer (person 
responsible for building ^the item) and the tester of thte item being inspecw: 
are also members of th($^nspection team. Managemiei^ does, not normally 
participate in. an inspeetiiB; 

b. The Process, ^jhere are five basic steps involved in. every 
inspection: planning, preparation, inspection meeting, rework ar?d- follow-up. 
The first inspection for* a particular item contains another step: overview 
presentation.- These steps are suamarized below. 




While these steps should not vary functionally for inspections^ conducted at 
different development phases, the responsibilities of the individuals on the 
inspection team will necessarily vary slightly. This occurs because the 
primary responsibility'- for the item shifts as the lifecycle nrbgresses. For 
^ example, during a design inspection, the designer is the focal point.** 
However, during a code inspection or document inspection, the implementation*' 
is the focal point. 

1 . Planning. Set up inspection schedule and assemble inspection team. 

2. Overview Presentation (conducted only for the first inspection of the 
item during the development process). Distribute applicable Product 

- ' • - " ■ ■ ' ■ • . ■ 
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'and Backup Documentation and present a high level <sumary of ^^item 
to be inspected. : ~ .'. ' ''W? 1 ^'-' 



3. Preparation. Team meobgrs read and review documentation .^d^I^t^any 
' ^questions. . ...... ,•..-■■•-.« .\ : ^V' :v ;" '^rW*^ • .. 

^-fiispecifon^^ 



errors detected. Use checklists to ensure inspection completeness and ~ 
: . Problem Definition ^ 

5. Rework. Estimate time -to correct errors and:^±^^^t the corrections. 

6. Follcw^up. VerifV that all errors have been corrected using Problem 
Def ini.Upn; Sheet as a checklist. Complete -Sunmary and Management 

• ' Reports. ■■ ■ ... r- 

4.16.6. Example. The following ijs an exapple of a design- inspection of a 
software ccwpcaaent or item which defines the roles and responsibilities of the 
inspection team members. Upon decision of -management . tcf conduct a^ design 
inspection, the selected leader initiates process planning by identifying team 
memfcfers and .their roles and responsibilities. If this is the first inspectidh 
for this item (i.e. , there has been no requirements ^inspection), the .leader 
next schedules an overview presentation. Tije project* and backup documentation 
(i.e., Functions Specification system flow charts, *etc.) are distributed and 
the item designer -leads the-team through a high lev.el description of the item. 

itfter l^ presentation each team member reads and - reviews the distributed 
documentation -and listsp any questions. This list of prepared questions is 
of ten givgn to the- letter arid/or designer prior' to , the inspection meeting. 

At the designer iaspegtion meeting the implegenterlleads the team . ^through a 
detailed description^ of the design of the &&n being inspects. Backup 
dCcunentation. facilitates the description* and clarifies jSoints whLai may -be 
brought up. The checWist is used by each team m£nber to help identify errors 
and enforce stjaiidards. - The problem definition sheet is, prepared by the team 
leader at the end of the inspection. « The item design will either be approved 
as-is, approved with modifications, or rejected. In the last two cases, . the 
problem definition sheet is given to; the designer and the correction- process 
begins. \ 

^ At the gtart of this rework process an estimate is made by the leader and 
"designer specifying time required fcSSrcorrection. This estimate is entered on 
the Problem Definition Sheet and is provided to management. Management can 
then make a judgment as to whether their project schedule wUMbe affected. 
Necessary changes to the item are made -and the item is either reinspected or 
; subqgtted to follow^uix procedures. »^ - 

During follow-up, the Problem Definition Sheet is u^Kfc-as ^. checklist for the 
leader and designer to verify that all errors have been analyzed and 
corrected. The reader then fills put the Stmnary and Management fteports and 
submits them to management. * 

- - ' v : \. . \ , * " . 

*ll6.7. Effectiveness. Since the cost torcorrect an error increases rapidly 
as the development process progresses, detection, of errors by early use of 
inspections is an attractive prospect. 
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studies have been carried out wtoch indiciate that inspections are an effective 
• ^ jnethodh of y- % increasing /product quality (reliability, usability and 
maintainability). Experience .With: the technique * ixi(U<»tes..^that : .-it->/ls' 
effective **on projects of all sizes. The. best results are generally- achieved 
when the Inspection leader is experienced in the inspections-process. 



j Sane of the best quar^ of the use of inspections have come frcci 

■IBM, ^diidi* hias been stuping thejuser-of : the technique; for a nunber of years. 
One study, detailing and cdnparin^ifce. benefits of inspections and structured 
walkthroughs, vindicated ' 23% hiiiier programmer productivity with inspectipris 
than with walkthrpu^is . No. data was ^available docunenting the amount <pf 
increiased programmer productivity attributable to inspections alone. The 
study also .reported 38% fewer errors in the running code than if solely 
applying walkthroughs as an error detection mechanism." 

.The. qualitative benefits attributable to the.' use of inspections are 
subs^tantative. The following list is- illustrative at some of these positive 
" effects: ' a » 

- ■ -if"- * * - "• " 

/ o. Programs which are less complex 
o Subprograms which are written in a consistent style, complying with 

established standards a 
o Highly visible syst^enis development " 
o More" reliable estimating and scheduling, / r 

p Increased education and experience pf^Ll iftdividuals^ involved in the y 

inspection process „ .;3:,tc?: " '^-Si'-f.-' • 

o Increased uSer satisfaction "." x * . ' & \ ■ ~ - 

o Improved documentation ' ~cT. '/ . : " * - 

6 Less dependence on key personnel for^^iti^* ^13^ \ 

4.16.8. Applicability^ While the most cqw^ inspections are for 

design and code, the technique is lifted to these phases and can be 
applied duringr all phases, for most type^ ^ .^^ications (i.e., * business, 
scientific, etcr)^©n • large or smaH projeste.'^ ife .A'h 



4.16.9. Learning.' The experience of the inspg^tioh leader is essential * to 
the success of the effort. A" correct attitude about the process Is* essential 
to all involved, including the appropriate managers. Many excellent texts 
about - inspections (and other types of reviews)\are in existence which should 
supply the required level of detail as well as discuss seme team psychology' 
issues ^>ertinent, to 'inspection conduct. / 

4.16.10. 'Costs. . The method requires no special tools or equipment. The main 
cost involved is -that of human resources. If inspections are conducted in 
accordance .with the resource guidelines 'expressed in most' references, the 
costs e pf inspections are negligible ccmparecuwith tiie expected^retuf ns. It 

^should be kept in . mind* that follow-up inspections :lBb correct previously 
detected errors can increase the original cost estimation. Most references 
-suggest that inspection meetings should last no longer than 2 hours, and 
reasonably be kept to 15 minutes. Preparation time can amount to^es little (as 
1/2 hour and should not require longer ^han 1/2 day pgr inspection. 
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4.17.1 ^ Name. : Interactive Test Aids. - ,^v^ : ; ' 

, 4. 17.2. jpSasic featit^s. . Interactive test aids, debuggers,, are xools . used to 
control and/or analyze the dynamics of a ^program during execution* The 
capabilities-provided by these tools are used tct assist in identifying and 



isolating program errors. These capabilittes^ allow "the user to: 



o suspend program execution at any^ point, examine -program status, 
V © interactively dunp, the Values of selected variables and memory locations, 
o modify the computation state of an r executing program, 
o trace the poritrol 

4*17.*3* Infonra^ Interactive test aids require "as input the source 

' code that is to be executed and* the commands that indicate which testing 
operations are to be perf ormed By the fcxfcjdtiring ex deluded in the 

ccmnands are indications of which prtjgr^ stal^ be affected by the 

tool's operation. Ccnmands can be ~lns^t^?in the source code and/or entered 
interactively by the user -during r J at preselected break 

points. ' w . .■ 

K 4*17*4. Information output, information output by ari > interactive test 
aid is' a 'display of requested infonnatio^ of * a program. 
This information may include the ^contents of selected storage cells at 
specific execution -points or a display of control -flew during execution. ' 

4.17.5. Outlinie of method. The ftjnctiqpa perf drmed by an interactive tesl£; 
aid are determined by the commands input to it. Some common commands 
described b^t«ir. \-..ur - ::--^."::. . 

BREAK: Suspend program execution i^erra particular ' statement is M y' 
executed or a particular variable is altered. . 

DUMP: Display the contents of specific storage cells, e.g., variables, 
-internal registers, other memory locations. €f 

TRACE: Display control flow during program execution through: printed 
• . traces of: " ^" 

o statement executions (using statement labels or line 

nunbers), ^ * . v^: ; ^ 

o subroutine calls, or p^*- \ , 
o alterations of : a specif ied variable. 

SET: Set the value of a specified variable. ; 

CONTENTS: Display the contents of certain variables at the execution of a 
specific statement. • . ; 

SAVE: Save the present state of execution. 

RESTORE: Restore execution to a previously SAVI 

CAII.: Invoke a subroutine. 






EXECUTE: -Resume program executiosfcat a BREAK ribint. * " '-- 

.EXIT: Terminate processing. 



These commands allow complete user control over the computation state of an 
executing program. It allows the tester to inspect or change the value of any 
variable at any point during execution. :..'■*■%,.} 

The capabilities of special interactive testing aids can also be found ^many 
implementations of interpreters and compilers for such languages as BASIC. 
. FORTRAN, COBOL, and PL/I. - ' ~ ; . ' 

4.17.6. Example. A critical section of code within a - routine is to be 
tested. The code computes the values of three Variables, X, Y, and Z, which 
later serve as inputs to other routines. To ensure that .the values ' assigned 
to X, Y, and Z have been correctly computed in this section of code, an 
interactive testing aid is used to test the code. 

- s ' • "«.'• ' 

Two BREAK commands are initially inserted into the code. A BREAK command is 
inserted innedfately ^before the firs^ statement and immediately after the last 
statement of the section of code being tested. To display the value of X, Y, 
and Z, ■ a CONTENTS command is placed before the second BREAK command. The 
progr am containing the above-mentioned code is executed. When the first BREAK ' 
command is encountered, execution is halted and a prompt is. issued to the user 
requesting that a command be entered. A SAVE command is typed by the user in 
order to save the present state of execution. Then SET command is entered to 
set the values of two variables, A and B, which are used to compute the values 
of X, Y, and Z. . The EXECUTE command is then issued- to resume program 
execution. ; . ' — . . 

At the end of execution of the relevant section of .code the preinserted 
CONTENTS command •displays the computed values of X, Y, and Z. The second 
BREAK command allows time for these values to be*?examined and gives the user 
the opportunity to enter* new commands, At this time,- a^BESTORE command is 
entered that will restore the computation state to tg^state that was 
previously saved by the SAVE command. . For 'this example/ oBomputation state 
returns- :to that which followed the first. BREAK command,' allowing the' code 
under' analysis to be tested with different, input values: Different values for 
A and B are entered. and the contents of .X,Y, and Z are observed as before. - 
This -process is repeated several times using carefully selected values for A 
and B and the corresponding values of X, Yj^land Z are closely examined each 
time. If- results of several computations look suspicious, their input and 
output values are noted and -the code is more thoroughly examined. The program ' 
is i finally terminated by entering the EOT command at one of the two possible 
breakpoints. y *, 

4.17*7. Effectiveness. To be an effective testing tool , an - interactive test 
aid should, be used with a disciplined strategy to guide the testing process. 
The tools can be easily misused if no testing methodology ;is combined with' 
their use. '' •" 
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: 4. 17*8. *' Applicability.- 'interactive --test. -aids can. be applied to : |Sy5fce ' of * : 
source code. J Most existing -tools, however, are language dependent^&W , vi%£t 
operate correctly only^orcsp^ified;langyiages). \ : r 

4.17»9. Learning/ A minimal:- 'apount :df - learning **is required to use .these 
'■ tools. It is: comparable to the learning required in using a text editor; 
-< JJowever^-if-:the tool-is 4xT-be used most-efficiency^ : 

in utilizing the tool with, an effective testing strategy. < ~?va£W.' - -- V 

f 4>17.io. Costs. frogi^amsj*^ "aid Twifi/.' 

^require more computing- re jjragg s (e.g. . execution time£ : memory for : di'ageostic 
tables) -than if executed uM rmai^&eration.r< ^g/ccsst -is dependent bn'theW 
implementation of the . tio3Km,06c 'example, ^those -biased on -interpretive^ 
execution will involve costapHlff^ 

.4^7.^.1 3?References. :*' f - ~- >y ' 
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4;18Lt. Name* ; Interface ChiMce^?^ 




4. 18.2. Basic IfjMtures.V the^ consistency and ' 

completeness of the'infrfiti^ between exponents, modules 

.or procedures . of a system. ■ ~ * . • :r , 



- a. a fbraal representation: of system" requirements "or ; : . '•X'^;^^'^ 

b*: a forina^ ■> 
. . -c^. a prb^a^bod€fa v to ! . e " - 

4.18.4. Inf qira interface inconsistencies and error^iare 

revealed. as terror messages included wit*^ U 

source listing* or as a serrate report..- ; ' ' \ ----- < .«/. I 

4.18.5. . OuUine: of method. Ihterfac checkers arfe: fully, automated tools : 
Which analyze a computed prc£essable form d£ a software system requirements 
specification, design- specif icalion.p^iDde. The method ~for each of the three 
representations — requirements* ' design, arid code — will be illustrated tifelcw ; 
by examining ihe interface checking capabilities of three existing tool£. < ?>' 

PSL/PSA CProbten Statement I^anguage/Prolstok ^5tet6ment : ;Anaiyzer) Tl) is ^an 
— — ited reqpbieite^ -describes 



stem rec^dwiients as->a system of ^ j^H^ processes ! and output; Both' 
information. ,an'd ^ control flow--sre r^i«|SBtwitoin PSL. " In t^ace- checking 

* performed ty -PSA c^g ists* of assuring Tthat^atl ^teta items? are^used incl 
generated ' .by> suiiifc process and tthat r-al^ pr oc«fes - use data ^ <ocomplete 
requirements. specilBation are, th^tfdrej- .i^ily/«wected. ^ 

-Ihe^D«ign Assertion Cbnsist^icy -Checker • OQ^CC) (2> - is % tool **ich> analyzes 
, nscdule interfaces based *on a design whicfcr cbnteihsV informal describing, for ~ 
eacdi jD^diile, ; the teture of s tfeV |: This iitf brmixion is 

specified' using assertio^^ to 

so "on-. DACC *: . 

: .bhecks module; calls ^against the assertibris ;: in ; th<e- calledV mbdiile Lfor " 
; r eonsisteh<qr. Thi^ produces a consistency >epor# indicating which assS^tions ^ 
havA been «&>latra. - : fc ' "y-ir- : - — i'"* ' ' 

'^0aT C3) Is S static analysis, tool fc which^ prisB^cS^/ used Tor checking 
j. Fortrai ^programs f or a*<toerence ...to a por^ble; subset * <tf ^ language 
-but; it also performs subprogram interface checking. PFORT mattes actual with 
dtjmny k ai-guments and checks? for unsafe references, ;suclf as con^raints being 
passeii as argunerit*. ') > * % ". 

. ; Interface Recking a particular 

.lahguage's compiler as well. , For '^cample, Ada (4) provides a. parameter 

• passing mechanism i*erel^^ranteters are identified to *be input pr output or 
> ; input/output. Moreover r ^lata type, and constraints (e.g. , range and precision) 

I must. match' between tHe. actual arguments and: the formal parameters (in - 
non-generic subprograms). 
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In sunnary, -int^a^ checking tools will generai^^Bi^ for: 



o modules **i6h are u?ed but not defined, >.•• :> m • V ; \ 
o w)dules">rtiich-are defined but iK)t used v 'V •^• V-.i ■; " .. 
o* incorrect nunber of arguments, V *:. r r ^ : ^ ^ 
o da ta ftype mismatxifies*between^acta^ and formal parjamgfcerysV > \ r 
o^ta-constrain t mis m atc he^betwe e n : a^ 



arr.- 



6 4&ta usage ^qn^lies. ' .' ■ ■ ; ;■ •., * W " * r ;/" ' . 
'.4U'1fc6.- Example. 



/ a. /Application. A StaUsticaL analysis package. "* wj^tten in Fortran ' 
utilizes a file access system J»* retrieve records containing 1 data used in the, " 



-analysis. ^ • ^ 



.X'a b. Error. :The primary record; retrieval stfa?outirie ^always passed a/ 
• statement number in the calling program which, is^to deceive control in, case ati 
^^bnonnal file processing error occurs. This i'£: the last .argument in..'* the 
-^rargumejg^lfst of -the ' sutxroutine call . - One program, howevery^fails ; to~supply 



-gumertt^ Tbe.cgljpilen is^iot able^to detect .the ^error: -Morepver, .-4' 
tor Jj0rfean implsmjtfitation ds slich^hat rip execution" 'time error . . 



theT: 

the _ ^ . 

occurs: unla^;r^^:tb^he : unsp^cif ied^statemeh^ "number ^s '. attempted, jf-ab* 
which time- th^systei crashes. . 4V .** " ' ' 

. . X^^^v,'^; : v : ,. •' -•> ; . ' . 

• J^-^'-^T^^^^T*'- T1 4 s error ..can easily* be detected. by»jjsirig • .\an*f|-i < 
iinterTace- cto^e^'^ phase (e.g.f 

PFOfi» of the.sof^r*e;developaient activity. Both. DACC.and PFORT "car* detect « 
incorrect hummers' of Sl^dments. ■ ■ . ; ; .v\3§ 

' ^ : ' * • • '• 1 ' '• ' ■ 

j#iiat»7.; -jEf f eetiveness*. Interface checkers are very effective at detecting a- , 
«laJss^~iof errors which <an be> difficult to isolate if leflf to testing. They 
arel g||erally more cost 
• as a , come 
specffSSltion toolv ; 




at effective if proyidedias a capability within another 
opiler 1 , ^ jJata 'flow analyzer or a requirements/design 



Applicability. .. The method is generally applicable. 
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■: 4*1^;9. Learning. \l!ne use of -interface checkers requires only a very mini mal 
. leartiing>effort. : r " : ' r ^ 

? 4.18.10. Costs. Interface checkers are quite, inexpensive to ^jjs^, usually 
much less than the cost of a ccmpilation. . ^ "\ 

jr. ' , i '* ''<c f v. . . .... ^ 

4.18.11. Refer«3ces.? • - 
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V (2) BOEHM, B. , McCLEAN, R. and URFRIG, D., "Some Experience with 
-.Automated Aides t6 , ^ Design of Large-scale Reliable Software". IEEE 
.T^aj^cU .1975 :(125-133) . • - . 
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- - (3) RYDER, B.C. and HALL, A.D.', " «The PfORT Ve^gier"!-! Canput^ng - 
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: ► ; "Preliminary Ada Reference Manual^ StgpiIan ngfac»* .- Vol. 14. No. 
:ffj&t A,'(June s 1979). ./ ^ : --^45Cv>-. . 
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4.19.1 v Name^^ Mutetiojjf^ysi's? : • !:./vO- - ~-*f|^ ' ' : ' - 

.4.19^. *• Basic features; Mutation analysis ' iP a" -technique 'for: detecting 
errors a program, and for determining, the thoroughness with 'which the^ 
prdgrani^as heen . tested. Oat entails studying ibfiW behavior of a large 
conection-Jof< programs which have been systematically derived . from the 

Information Sjfate&'Ih.e basic - input. requiefed by mutation analysis-: is 
source program, and a collection .of test data sets a£ whicfcthe 
ites correctly,, and which the user considers to adequately— and 
test the -program. , , .• '--v':. ' •.. ' .4 ^ 




-jar 



; 4.19.4.. Information outputs. Ihe ultimate output of mutation- analysis is" 
• collection -of test data sets and" good assurance that the collection is^i# 
. adequate to" thoroughly test the program, it is important to ittaiersta^P tdsl 
the mutation analysis process may very well have arrived at this- final state 
only after haying exposed program ej^ors and inadequacies in the xOriginalX test 
data set collection. 'Hence 1 ,;," it i£ not unreasonable tp cohsietifv errors. 
- detected, new program* understanding/ and actional test aata' sets /t?llso be 
information outputs of the mutation analysis? process. 

" " - ' - **• ' '• - • . . 

Hie essential^approach taken in the- - t mutation 



Outline of ^method. 

of a pro^ramj&t^^ of y 

ved from a trivial ' traosformation of -the : original, and ^to 





; ; .eacte~versiOh to- testing by theffciven collection of test data sets. 

^t^^natiire of/ the- ^transformations, - it vis expected that thfe derived 
/wi^l ^\Ms^\x^y '6iff^r^t programs** ran the original; Thus, the testiW 
regimen-should demonstrate^ that each is in .fact different. Fai^jre to <td- % so' 
invites suspicion tftat the collection ,of^test data' sets is •inadequate. This> 
usually leads to greater understanding df f^^ jscogfain and either* the* detection^ 
of, errors or an improved collection of telsti d^^Hets, or both. ~ s : ;- r -.';^p I *• 

centra^. fea&Ire of mutation analysis is l£e mechanism for creating the 
program mutations the ^derived versions of the original program.- Ihe set of 
mutations which is generated and tested is the set of all programs- which 
differ from the original only in a.aball number (generally "T or 2) of textual 
details, such as a change in an operator, variable^ or * constant. : Research 
appears to indicate that- larger numbers of changes contribute little or- r ' 
additional diagnostic power. \ 

. The basis for this procedure is the *tanpetent-Frogrammer n assumptions which 
state that progr am r errors are not random phenomena, but rather Jesuit from 
lapses of humaWTOemory^or concentration. Thus, an erroneous program. should be 
expected to differ from the -correct one fchly in- a\mall number of details. 
Hence, if the- original program is incorrect, then the /set of all programs 
created * by making a small number of the small textuar changes just described. 
should include the correct prrogram. - A thorq^^j^^eation^ofv^test data sets 
wcidd reveal r benaviqra^ 

'aikl the derived qorrecWenei ." .-. . " 




Hence, mutation . analysis entails determining whether - each mutant : behaves , 
.differently from the -original. " If so, the mutant is considered "incorrect. : Jf 
not, the mutant jnust be studied carefully. It is entirely, possible that the/ 
mutant is in fact functionally equivalent to the original program. If so, i£sr 
identical behavior is -clearly benign. V If not, the mutant is highly 
significant, as it ce rtainly indicates an inadequacy ,,in the , coll ection of test 4 
data sets. It may, furthermore, indicate, an error in • -the* original, program - ^ 
which previously went undetected because of inadequate testing.- Mutation # 
■* analysis facilitates the detection of such errors by automatically" raising the ' 
probability of each such error- and then demanding justification for -concluding 
that each has not in fact been committed . Most mutations quickly : manifest : 
different behavior under exposure to any reasonable £est data s^t collection, 
and .thereby demonstrate the absence of the error corr'espondingjbo the mutation 
by^^teh they were created. v This forces detailed attentiouson "those mutants 
whl^fbjfeave identically to the original and Jaiu's. forces attention on any" « 
abtag|§aerrors. . .• \ v 

< If ali JButatibns of the original pro^ftreveal different execution • bjehavior,* 
*; then the program is considered to pWa^quately tested and correct =within the 
• limits of the "Competent. Programmer """assumption/ ; — -> „ v : : * 

4.19.6. Example. Consider the Fortran program, figure 4.1 9^.6^ ? l*ic^icoim^^ _ 
•the number of negative and non-negative, numbers in array ..A:*T % ■ A . : 

|fe J- * SUBROUTINE COUNT (A, " NEG, NONNEG) ... ' -' > A" . 

'I.-*. \ -"DIMENSION A(5X ' . •/*• • v' :> •• . ,/» 

' V ••• : $ ■ NEG=0 ' ? . ••• - ■ ' ■ ' '* • ' ■• - * 

'■: nonneg=o« . 




'f -IF.iCA( : I)^GT.O) N&IBG^SRfflS^''. ^ 

• iF^a).LT.o) nbg=neg+i ^ • v.<- V 



• . . RETURN . - 

Figure ,4.19.6-1 Subroutine" Count , .*•',:■•> . 

• and the . collection of test data* sets produced by initializing A in turn to: : 

• " I - II III • > • 



-••••>"• 1 • 1 -1. 

• •/ .-2 .. : ' 2 ' -2 :■ 

-4 4 ~ -4 . X 

• >5 5 ' -5 '. •" . 

..'v •' .; - •; , | '. ; '■ . . 

Mutants might be produced based upon the following alterations: / 
a. Change an occurrence of any variable to. any other variable, e.g., 
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A tO: I 

NQNNEC to NEC 
I to NEC 




b. Charge- £n occurrence of a 
\. valuer 

; e.g., to-fv. •• 
••• - 1 to .0^,. • . • •• 
,0 to 1 
0 to -1 
.'1 to 2 




it to another ? cbnista*A whi 




is- close in 



c. Change an occur: 
. e\g.,*~ v " 
-iNECi+^ to NEC « 1 < • 
1 NEC + ..T to , NEC - 1 
Jtf0 .Gt.O to ACT) .GE.O t 

i*. ta).LT*0 tP AdX.NP.0T T 



of an operator* to: aritlbherrrOperator : 




ib^/'t^e/set'of ail "single alteration-" mutants would ^ consist «T all 
Gontedhirig^ exactly one^^pf the abone cS^ges,/ -The/sety o£ al 
alteration" mutants wbulctionsist of all prog^aSs" containing r a, ^ 
above: changes. <• " '. \ "-Vr; -v.-" ■ c t - > 



.Clearly^many siwli niutatiorte are radicap.j^^ffer«nt and would ;quicSiy manifest 
*5byioui^ rdif£^erit behavior. For example, in changing variable, Ivto A Cor 
toe prpgram Is gendered liicanpilable by most compilers. * Similarly 
cbS^b^ ^NBG=0" to "NEG=1 " • causes a different outccme v for test ^case I. ; 

Significantly i changing; ^ A(I).GE.O or A(I) .LT.O * to - £(I) .LE..0 

produces no- difference in rd^i&m behavior on any of the thjfe test data 
sets. Hiis rivets attention on "these mutants, and subsequently; ori^fcbe Issue 
of how to count zero entries;^ One rapidly realize? that the collection of 
test data sets was inadequate in that Jit did not include any • zero 
values. Had it included one, it would have inc&fcated thafc: """"" 

/ : IF (A(I).GT.O) N0NNEG=NQNNEG+1 shbuld have been 
IF (ACD.GE.O) NWINBGsNQHNEG+1 . . 
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Jlhu?, mutation analysis has pointed i but ,botfr this error and this weakness 
the; collection of test data sets.. After cfaangingytfae program anjt 'dbnection, 
all mutants will behave differently strongly raising our confidence in the 
correctiies^. of ttfe. program^ c.-frf . ^ > 

4.19.7. Effectiveness. Ifcit^ technique for 

defecting errors, but .it must be understood tliat it requires combining an 
ltfuk feuman with gocxl automated, tools.- Even then it mpst be understood 
\* it * is;' a IreL iable techriiiqae f or ~ dempps^atihg the absence only if all 
possible mutation errors (i.e., those involving: alteration, interchanging,- or 





mls^MfpP- m operators, variables,;, etc* ) are a ex^ 



The need for.\good tools is- easily "understood wto 
. prt^ram has- an enormous nunber of mutations, each" 

ex^i?ttsed:by^the test, data. setfc^ and evaluated'." Gn _ 
; appear to ^entaH ^thousands <tf^^ 
^toals-^ve— ^ 

~ representation ctf the original program. TZfXs repine 





ieaCLizesv 

must be generated, - 
face, tibis would 
[executions^ Clever' 
s^i^l^tnteraa^ 



^ . ,. - w ... .^^.^.JL oh is readily and 

efficiently transformed into the various mutations, and also serves as the 
; **S* S for very rapid Simulation 'of the mutants' executions, thereby avoiding 
the need f or compilation and loading- of each mutant. " 

vlfcis ^tpol'set still * does not typass the need for humans, however. ! Hunans must 
st^|> carry* .out the job of scrutinizing mutants which behave identically to 
the crigihal program in order to determine whether the mutant is equivalent or 
whether the a^r 1 "^ ~ ~ " * ' * 



Lection "of test data sets is inadequate. 



XT- 



At the end of . a'^uc^essful . mutation tt analysis^; many errors .may have^een 
uncbxered^ and .the collection of test; data sets^has certainly been made Very 
tfiorougfti Whether the absence., pfi errors has been established; however, must 
be, considered relative' to the "Compj|^t; Prograipmer" assumption. Under this 
^assumption, clearly. alJh^errors " ofTmutation *are detectable by mutation 
analysis;,- thus, ♦.the absence* of diagnostic messages or findings indicates the " 
rabseifce of these • .errors. > Mutation.' analysis cannot, ^however, assure the 
: 'absence df errors Which cannot be modeled as mutations.^ - . - A . '*' , 

* 4.1^8." Applicability; ; • Mutation anafysis is app^ently'^plicable to any 
" ^gorithmic* .sdlution ' specification. M previously indicated, it carrBnly be 
^hsidered effective when supported .by a body of sophisticated tools. Tools 
enabling analysis .of Fortran arid ; COBOL source .fcgfct exists; There is, 
- .fur^ermore, no reason why tools for other coding languages, as well > as ' 
^-algorithmic design .JLanguages3N«>uid not be built. ■-. ? u . 

Le ^ r ^^«* This^tec^que requires the potential mutation analyst to 
T>ecome familiar with ttie£^g|lbsophy and" goals of this novel approach. In 

additional t appears that th^e^e familiar the analyst is. ; with the- subject 

algorithmic solution specifications, the n»rVeffective|pe analyst will be. 

This is because the analyst may well have to analyze a collection of test data 
sjgsets to determine how to augment it, and may 4ia»e to analyze two prc>grara to 

determine,- whether they are equivalent., w.-'- \ 

4.19.10. Costs. In view of the previous discussion, it is important tb 
recognize that significant amounts of human analyst time are likely to be 
necessary to do mutation analysis. The computer time required is not likely 
to -be excessive if the sophisticated tools : described earlier are available. 
The interested reader is urged to"^ consult \the. following references for 
Lon of this. » / 

: * " ' ■ i • ' ~ 

References. ■,•■/•- 4 
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4.20.1. Name.. Peer -Review- 

4.2Q.£i Basic Features. A peer review is a process wtach proj>c 
personnel perform a detailed study and evaluation of ccxtej-docufientation; 
specification. Jhe term peer review refers to product evaluations^wtacft 
condticied Tgy- individuals of equal rank,, resppnsibil^^, ^r jDf s; 



xperience-and-^kil3^^ 
the , overall category of a peer review. Code 
walkthroughs and inspections are examples of 
formality, participant roles and responsibili 
rjeqyuired. 





swhichfvfs 
B,vii£>und-robin -reviews, : :f 
lews which differ . in£- 
put produced and inpu^^ 



peer review will vary 



4.20.3. Information input. The input to a parti? 
ijsiightly depending on which form of -/peer review is being conducted". . Ih 
general, each of the forms lof p^£*rlevie& require that seme sort of review 
package is assembled >and disttfitgrt^d. This package commonly contains- a^ 
sunnary of the requirement(s) ^ctf^are the basis for the " product being 
reviewed. Otber> common inputs are differentiated by the stage of the 
lifecycle currently in process.., For example, input 'to a peer review during 
tiie ccKling phase would consist of program listings/ design specifications, * 
programming ^standards arid a sunnary of results fnta „ the design peer review 
previously held on the same product.- Common input to. particular forms of peer 
review are described below;; (A sunnary of the methodology for each of these- 
-fcreviews,. appears In Section 5.) , - 



1. Code-Reading Review. 



o Component requirements 
o Design* s^i£ica tions 
o Program lidtffijgs. 
o PrograraningSstandards 

b. Round-Robin Reviews. 



o Component requirements 
o Design or code specifications 
r *9. - JfcQgran listings (if during coding phase) 



"4 ^^ff^F 1 ~ J \ ' 



^tP** . Walkthrough, 
o Ccmix>nen^ 

o Design or code ^Jecifications* \ ^ 
o Program listings (if coding phase walkthrough) ' 
Product standards - \ * 

tck-up documentation (i.e., flowcharts, HIPO charts $ 
^te/^tionaries, etc.) 1 
QJpuesjtion^list (derived participants prior to review) 

■ - *d. Inspections. 




o Program listings (if during coding phase) . « 
o frodudfestandaras , 
- o feck-«^ocumentation I ■'.■/• 

' °l? ie SH»^— ntainin 6 descriptions of particular features - ^ 

v yto l^gBguated) v - . 

• ^^^Saon output. The output from a peer review rvariesfebnn of - 
—review. cne output connion to each form of a peer review is a^isinn 
, con^nsus^aboutt&e product under review. . Ihis 1»SSm5 fe^Sg^S \ 

£2&^? PrOVal 0f ^e - product as. is, an - approval rSSaDendel 
modifications, or a rejection (and rescheduled review date). ctanenoed 

* Specific output frcm^peer reviews. described in Section 5 *are^ ■ ~.1k 

- a. Code Reading Review and Round-Robim Review. 

o Informal documentation 'of detected proems " f v ' ■ 

o Reconmendation to accept or reject reviewed product a 'rV^ -i"*- 
. o Discrepancy List •; •. • ' ' 



b. Walkthrough. '• .;' \- ^•--';--v* ; ^^^/ : - ' " * 

o Action List (formal documentation of problems^ - . ' 
^Walkthrough Form (containing review sunmary and group: decision*) 



✓5 



c. ^Inspection. ■•" tt . "' 



1. w 

3* • !• v 



^P^fe 30 -^ 6 *^ ^ Memo (defining individual roles 
f# '^^^^^'es^agada and schedule) • j 
^ ftttblem i&ihitjgEr Set- - -- <r 

Sutoary ' report ^documenting "em>£ . 1 

• — . related statistics on the ertfcrsW * - - . • > 

° statSs)^ reP ° rt ' eVrors, problems and component ^ 

iT S£ Pl *' m ? st ^ r reviews are not attended by management/ (An exception 
££L in circ i? stances where the project manager, is also a design^, coder 
Z ^fw-r """W °\ very 311311 Projects.) The presence, c^agS** tends 
to inhibit participants, since thfy. /fee*. - that: they areloSSnSy bSn? 
evaluated ihis would be* contrary to the . intent of ' pS^rtES^SPSa? 
studying the product itself. . . . • ^ reviews — vnat oi 

Another common feature is the assembly and distkbution ^f " prejec^ review* 
materials prior, to the conduct of the^er review. This . aliows?ParWc$bants 
to spend seme amount of , time reviewing ttoS oata to bk^j^^^^^^ 



v - 93 ^ ^ 



0 
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:y,: 



£^|f£^ decision about the 

t^gji^ review: product. This decision is tisua^ly connunicated to 

management. %?■ . ■ 

MOst rwieiife,^ in a^group organizatigp as opposed to individually 

by^i^ project team itself . * While t^s may seem an obvious 

i t bears seme di scussion. 



. _ w software- 

dev^pment and/or maintenance empl<$\ seme variation of ^ team approach. Seme 



M o st organizations — doing- 



T^share 

leads the 

a. , 

secretary, 



team organisations are described t>elow. 

o Conventional: Team ^ A senior progranmer directs the efforts of one or 
more less experienced programmers 
\ * ^Egoless Team^Pro^aniDers who are of about equal expert 
v pnoduc£ ; rj^ponsibilities. ^ 

o Chi^^o^aBDmer Team*- A highly qualified senior progr; 
■■ • ■ eff<^t^^ 

:. res^^bilifies have tjeen . assisted (ije. , back-up pr< 
l|t4^^^^|&.>. ^ 

» The group, review is^jipt necess 

m the team <^^^2e^ to manage ar^^omplete the software pre 
* grbup is lik^Sfc^^e composed of a subset -of the project 
individuals J r ^^9^ed by. the form of review 4>eing held 



lifecycle iik; 

attained 
^responsibili 
filSeloww T& 
'♦tfist. 



- '3- 




y the same as 
t..\ The review" 
v plus other 
_ e- stage of the 

s. "Hie -benefits of peer reviews atp£ ^vtinlikely * to be 
e. group acts separately, withoirt,^ - designated 
ie rele^lgg^nly used in - review jgro^^: are ; described 
are^:^^^H employed i^^ny one revietr but represent 






q Group/RevidS-^^ by management with 

^planning, detecting, organising and coordinating responsibilities. 
■Usually has responsibilities after the review to ensure that 
recommendations are implemented. 0 * 
Designer - |fee individual responsible for the specification * 
of the s >roducjt> andj^^J for itS/ in^l-^entation. 
Implemetfter - the xfioividual respbhs^t3Bi^f0r -developing the product 
according to thei 'plan detailefcf by ttfe'designer. 

Tester - individual responsible for testing the product as developed 
by the implementer. 

Coordinator - -the individual designated with planning, directing, 
organic j^^pd coof^dinating responsibilities. 
Producer^-^the i^wividual whose product is under review. 
Recorder^- ihe individual responsible for documenting the review 
activities during th'e review. n 
* User Representative - the individual responsible for ensuring that the 

user's requirements are addressed. :\ * - 
0 Standards Representative - the individual responsible fori 'ensuring that 

product 'staflttards. are conformed tot,*..**...: v * 

x>: l^intenance Representative - the ii^yidual who will fe' resjpohsible - for 

updates or corrections to the installed nj^iuct. : 
o Others - individuals with specialized skills or responsibilities which ; 
contribute during .<tjpe peer review, ^ - * 



o 
o 
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While the forms of peer reviews hav 




aSS«on 1 Participant roleVand responsibilities,, thefare^ff er^S V 
application. The remainder of this section will summarize the aubSStiS - 
me^ associated with, the forms of peer reviews preSy introduced' ' 



• - - - •• . a. - 
evaluation 
which .has 



Code Reading Review. Code readigg is line-by-line study and 

been compiled and is free of .-syntax ert-ors. However sane 
organizations practice code reading on unfiled iwce listtSfor -£5 
written code on coding sheets in order to remove syntar^Wife^orl pS 
to <»de entry. Code reading is ccmmonly practiced pn^^^Str^ctured 
3gode and becomes cost ineffective when performed b^stru^^d^code 




<S>Jr 



The^optimum size of the code reading review- teaPW three' to four She 
producer sets up the review and is respSibleWteam leadership 
three programmer/analysts anT selected by^toe ' product Ssed^Son ££r 
egerience, responsibilities; with interlcing prg^br^lr • 

Die p^xlucerVd^tributes the review input (see section ^20. 3> about two"' 'datffe 
UrJT^: ^J? 8 *^ rr iew *e producer: and the reviewers go 4o^ eS ^ 
1^ of code ^ more^refdable '-" 

•^r^^rt^f^ reaShgf S^be ' * 

Pertormea. readgng fopunderstanding andlreading f or^Hrif ication ^artir«r 

^^^f^ «<We works.- its structure, what functions it ^3SS? 

dSicSISe - "'^CSSf^ft^ - s ^ ldar ^ ^un&g that figure^-H^f 
aepiccs ^the .structure ... a^-program v ccmporient, v a reviewer readiar 

f.i,. 2-2, 3.0, 3.V 3.^3*3. ■ v;v: •/ ; \ * 



2.1 




Figure 4 .-20 .5-1 A Program' S^eture 



bot^f^^/J 0 ^^^^^ reading for verification implies a. 
^ STroT^i ^ ^ he , c ? de ; 3e component depicted above would be perused 
£rZZ or **l; 3-3, 3.3g3,1, 3.0, 2,2, 2.1, 2.0," 1.0'.. if this 
manner^ it is possible to produce, a dependency list detailing para^eterT 
control switches, table pointers, and internal and external^arSblS^wl; 
the component. The list can then be used to ensure hierarc^ical^consistency , ^ 
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data-ayailabnfty variable initiation etc* Reviewers jx>iht out any problems 
Jor- errors detected while reading for understanding or verification ^during the 

o^-fceview. ' -X--;.>-. .*-*•' - ~ . ■ ; ; * 

g|The team then makes ian informal decision about the acceptability of* the-'^e 
^;.c product and may recommend changes.- The producer note^ suggested nxftificatibns 
: ,and is responsible for all dianges to the sou rce code, Suggesfagg^hanges .-."are 
•jgyaluated by the producer and Vieed not be implemented f ^^i\e^ producer 
^ determines that tfaejf are invalid. - v - v J^l*. i 

Jhere is n.o mechanism to ensbre i^iat change is inplementej br S^oilow up „on 
the .review. . - ]r<- -V- •. " \. - \ : r V^<"'- 

; b. ; Rpuntf-Robin Review. ; A round-robin review is a peer review wfaerg- 
. each participant is given "an ejqual„and similar share of the product being 
previewed to study;, present', -and lead : in its evaluation. > ;.;.:*•* 

' ^ round-robin review can be given <iuring any phase'bf thV product lifecycle 
is' also ' ; useful for • o>eu^entatiofi-» review. In addition, * there -are 
;<;i/ariat$Qns :6£ the ■ rca^-robin review which incorporate seme of thea best 

'f features* from • '. other peer . review forms, but continue to use the--alterSating* 
v -review leader" approach.^ For . example; 1 during a round-rob'i$ inspection, 'ea&h 

* ^item-VvQ|i>the inspection checklist is made the responsibility of alternating 
'T^tieipants, . ; ., i' "y l : : \ : -Hv." ... W K ' : -i. **.'v'..V 

Y ■. . - '■. .' ; : •:■ "lift/ '"■ " ' '"\ ' 

ooasboti' nunber of people involved in. this type of peer, review is four to' 
-r^six*: - ^Ihe fleeting is sch fcy^ the producer, who also* distributes *scme high 

level documentation as described in section 3 ?5.The producer *wHlx either be; 
the -first review leader or will, assigft: this r'esjporisibiiity to,.another j 
. ^ participant • The temporary leader will guide the dther participants <kio may 
be implementers, "designees^ testers, users, maintenance, r^freisentatiyes^ etc. ) 
the first unit of work.. Biis unit may be. a nKxlule,\paragrai^/Jr 
^code, 0 inspec.tibh xtm, 0£ other unit of 4nanageable^ size. - ^'102 participe&tsV 
X including th^^leader), ^^ 'the opportunity to conpent^i^uthe xiiit bef ore^?^bhe 
next leader begins fthe evaluation^ of the next unit. The : leaders ^are 
v respcmsible for noting maj or comments, raided, about their pi ^e: of -itorki> At 
\ the end of the Vev^iiew. iall the j^ 

decides whether brjhbt to approve -the prbducti 4Jq formal mechanism -for review 

• follow up i^tSSed, y^j A ' ! ^ v",;^ ■ ; f /- : > 

" * wali^^ouglis^ilhis'l^^^ of peer "review -is -%iore formal than the 



•: ' .»r> • rourid^biife /T review . k District' ' roles *•. and 
^.re^pa^bi^tstes are assigned /prior t&— review. Prereview^ preparation " is 
l" : 7grpie?*$^^d : a -more,- •'•jQormal' ^approach to problem documentation is stressed. 
^Ji^^^erlieaturV^o^ -by the producer. 

"^|Bg^i^^-^cdiBpn' walkthroughs are those held ...during design and code, yet 
f '^siejBtly they are betog applied to specifications documentation and test' 
= r^sults\,' ^ ■* . *- . •••••• ■ 

: ;The^producer schedule^ the review, and assembles and~ distributes input as 
' 'described in f section^ i. ' In most cases the producer selects tlie walkthrough . 
- participants (although sometimes this is done by management) and notifies* them 
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of their, ; roles and responsibilities. • The walkthrough ^is, usually conducted 
wjih less than seven participants and lasts not more , than 2 hours. If more 
time is needed a break must be given or .the product should be reduced in size. 
Roles usually included in a walkthrough are producer, coordinator, * recorder, 
and representatives of user,- maintenance and standards organizations. 

* - ' ' 

The review is opened ' by t the coordinator, yet the producer is responsible for 
leading the group through* the product. In the case, of design and code 
walkthrough, the producer simulates' the operation of the component, allowing 
each participant to comment based upon his area of specialization. A list of 
problems is kept and at the end' of the review each participant signs the list 
or other walkthrough form indicating whether the product is accepted as-is, 
accepted with recommended changes,' or rejected*. Suggested changes are made at 
the discretion of the producer. There is no formal means of follow up on the 
review comments. However, if the walkthrough review is" used for products as 
.they fvolve during the lifecycle (i.e., specification, design, code and test 
walkthrough), comments from past reviews v can be discussed at the start of the 
next review. * 

d. Inspections. Inspections are the most formal, commonly used form 
of. peer review. Ihe key feature of an inspection is that it -is driven by the 
use of checklists to facilitate error detection. These checklists are updated 
as statistics indicate that certain types -of error are occurring more dr less 
frequently than in- the past. "The most commonly held types of inspections are 
conducted on the product design and code,, although inspections, may .be used 
during any lifecycle phase. Inspections -should be short since they are often 
quite, intensive. This means that the product component to be reviewed must be 
of small size. Specifications or. design which will result in 50-100 lines of 
code are normally manageable. This - translates into an inspection of 15 
minutes to 1 hour, although complex components may require as much as 2 hours. 
In any event, inspections of more than 2 hours are generally "less- effective 
and should be avoided. Twq or three 'days prior £o the inspection the producer 
assembles input as described in section 3 and gives it to the. coordinator for 
distribution. Participants are expected to study and make comments on the 
materials prior to the review^ 

Ihe review is lead by a participant other than the producer.- Generally, 'the 
individual who will have the greatest involvement in th£ next phase of the 
product lifecycle .is designated ' as 'reader. For example, a requirements, 
inspection would likely be lead by a designer, a design review. by an ' 
implementer, and so forth. Ihe excgptiop to this occurs for a code inspection 
which is lead by the designer. Ihe inspection is organized and coordinated by 
an individual designated as the grotip leader or) coordinator. 

Ihe reader goes through the product component; using the checklist as a means 
to identify ccmmon types of errors as well as standards violations. A primary 
goal of an inspection is to identify items which can be modified to make 7 the ' 
component more m understandable, maintainable, or usable. Participants 
(identified earlier- in this section) discuss ^any issues which they identified 
in preinspection study. ) c ' 
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At the jpnd of tlhB inspection an • 'acbept/re j ect decision is made bjr < the oup 
and thei coordinator sunaiarizes : all:; the errors and problems detectoM £nd 
provides thii list^to all>iterticipants/:: The individual Jwhose work was (under 
-review (designer,; implemented j tester, etc;) uses the list to make revisions 
to the component. ' ' Wher^ revisions ' are ' implemented , " the coordinator and 
producer go through a minireview using the problem list as a checklist. 

The coordinator then^ completes Management and Siianary - Reports. The Sutnnary 
report is used to update checklists for; subsequent inspections. 

4.20.6. Example. The following is; an - example describing a code reading 

review..-' : ^ . ■. '.-W " _ --\ ' ' ' • v v . ; ' \ - : 

Three days prior to estimated ccropletibn of coding, the producer of a program, 
component begins preparation for a code reading review.. The component is 
composed of 90 lines of FORTRAN code and associated comments. The producer. 
• obtains copies . of the 1 source listing, and requirements and design 
specifications for the component and distributes them to three peers, 
notifying them of the review date and place. 

Each, reviewer reads the code 'for general understanding, reyiewing a major; 
function and >its supporting functions p prior to reviewif^ the next major 
function (see section 5). - 

One reviewer notes an exception, to the progranming standards. Another thinks 
that the data names-are. not meaningful; The third has found several comments - 
which inaccurately present the function they describe. Each reviewer makes 
a . note of these points * as well as any comments about the structure of the 
component. Next, .the requirements are studied to ensure that each requirement 
is addressed by the component. It appears that the requirements have all been 
' met * \ 

The . code reading -review is led by the producer. After a brief description of 
the component ancf its interfaces, the producer leads' the reviewers through the 
code. Rather than progressing through the component from top to 'bottom, the 
decision is made to perform code-reading from the bottom up. This form of 
code-reading is used to verify t£e component's correctness (see section 5). 

As the code is being perused, one of the reviewers is made responsible for 
keeping a .dependency list. As ^each variable is defined, referenced, or 
Modified, a notation is made' on the list. — 

The verification code reading uncovers the use of a variable prior to its 
definition. This epror is documented on an error list by the producer. In 
addition, each of the problems detected earlier during the code reading (as 
performed by each individual) is discussed and documented. 

At the end of the review, the error list is summarized to the group by the 
producer. Since none of *the problems are major, the participants agree to 
accept the code with the agreed to minor modifications. The producer then 
uses the error/problem list for reference when making modifications tp the 
component. 
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4.20.7. Effectiveness. Studies , have been conducted which identify the 
following qualitative benefits the forms of peer reviews. 

l - . 
' o higher status visibility, . 

• o decreased debugging, time, '* . . - * 

o early detection of design and analysis errors which would" be much more 
costly to correct in later development phases, 

o* identification of design or code inefficiencies, 
o^ ensuring adherence to standards, 
d increased, pfogram readability, 

o increased ufcer satisfaction, • 
o conmunicatian of new -ideas or technology, 

o increased maintainability. * • j 

Jt^ y available which identifies the quantitative . benefits 
attributable to the use .of' a particular form of peer review. However, one 
source estimates that the number of errors in production programs was reduced 
by a factor of ten by utilizing walkthroughs. Another source estimates that a 
project employing inspections achieved 23? higher programmer productivity than 
with walkthroughs.. No data was available, indicating the amount of increased 
programmer productivity attributable to the 'inspections -alone. 

4.20.&. Applicability. Peer reviews are -.'applicable to large or small 
. projects, during all development phases and ."are not limited by project type or 
complexity. \ * ■ , 

• - • ■ . 

4.20.9. Learning. None of the peer reviews discussed require ' extensive 
training .to implement. They do require familiarity with the concept and 
methodology involved. Experience has shown that peer reviews are most 
successful .when the individual with responsibility for directing the review is 
knowledgeablejabout the process and its intended results. 

.4.20. 10. Costs. The reviews require no special tools or equipment. The main 
cost involved is that of human resources. .If. the reviews are conducted in 
accordance with the resource guidelines expressed in most references, the cost 
depends upon the number of reviews required. Most references suggest that 
peer reviews should be no longer than 2 hours, preferrably ~'l/2" to 1 hour 
Preparation time cap amount to as little as 1/2 hour and should not require 
longer than 1/2 day per review. • ^ 
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4.21.1. Physical 'Units 'cheeking.'' - : ' 

• nlS-w f? ature 5- • "any (scientific, engineering,, and -control) programs 

perform conputations whose results. are interpreted in terms of* physical units, 

*2S5iS 2^ met f S ^ Wa ^'' 31,(1 joules - Physical units chSlSg^eSS 
tf^? f 1C S fcl 2" ^ °r in .program computations, in a manner 

similar to dimensional analysis. Operations between variables which are not 

.commensurate, such as ^adding gallons and feet, are detected. 

4.21.3. Information input. ..: Units '• checking requires three things to be 
specified within a, program: 1) the set of elementary units used (such as 
leet, inches,- acres), 2) relatipnships-between--the elementary units (such as 
feet = 12 inches, £cre = 43,560 square feet), and 3) the association of units 
with program variables. The programming language used must support'* such 
f specifications, or the program must be preprocessed-by a units checker. 

4 * 21 A .^ oniiat:Lon output. The information output depends upon the specific 
capabilities, of . the language processor or preprocessor. At ^minimum, all 
operations* involving variables which are not commensurate are detected and 
reported.^ If variables are. commensurate, but not identical (i.e., they are 
the .same type of quantity, such "as . units of- length, but one ''requires 
application of- a scaler multiplier to place it in the same units as the 
other), the system->may insert the requireti.multiplication into the code, or 
- may only report what factor must be applied by the programmer. % 

4^21.5. Outline of method. The specification of the input items" is the 
extent of - the actions- required- by the user." Seme systems may allow the 
associations a units expression with an expression within • the actual- 
program. Thus, orie may write L0TSIZ£ t . (LENGTH * WIDTH * square feet) as a 
boolean expression, -where the product of LENGTH and. WIDTH must be in units of 
square feet. The process, of ensuring- tfcat LENGTH * WIDTH is in square* feet is ' 
the responsibility of. the processing system. - 

4.21.6. Example. -A short program in Pascal-like notation is shown, for 

area -of a right circular cylinder. The 
program requires as input the radius of the. circular base and the height ' of 
the cylinder. Because of peculiarities in the usage environment of- the 
program, the radius is specified in inches, the height in feet; volume is 
required in cubic feet, and the surface area in acres. Several errors are" . 
present ;m the program, all of -which would be detected by the units checker. 

In the following, comments are made explaining the- program^ the errors^ it • 

contains! and how they .would be detected. The comments are keyed by line 
number td the program. ' J 

JLine^liuifcer Ccnment 

2 All variables in the program which are quantities will be 
expressed in terms 6f these basic units. 

3 . These are the relationships between the units known to the 

units checker. • t 

5r10 .. Variable ra'dius is in. units of inches, height is in units 
of feet, and so forth. 
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,12 Input vaLues^are read into, variables .'radius and height. . 

\ 13 J-ateral surface must be expressed in square feet- < RADIUS/ 120 

•''*'•. js in f£et, and can be so verified' by' the checker.' 

V5 - r Lateral-surf ace- and top-surface are both expressed in square 
. feet, thus their sun is in square feet, also* Area is 
•' * expressed in acres, however, 'and the checker will issue'. * 

a message to the effect that though the two sides are- ' ' 
• * commensurate the conversion factor of 43,560 was omitted '*' 
from the right side of the assignment. 
* 6 Ihe^checker will detect that. .the. two sides of the assignment 

arfe not commensurate. The right side is, in units of feet 
• • quadrupled, the left is in feet cubed. • . * 

■ (1) program cylinder (input, output);. ' • - • ~ 

(2) elementary units inches, feet, acre; '*- V 

• (3) units relationships feet - 12 inches; acre'= 43,560 feet**2; * ". 
(4) constant pi = 3.. 141 5927 ••»...- 
- (5) var radius (inches), ,• 0 : 

' (6) height (feet), ' ' . 

(7) volume (feet««3), ' ' . 

(8) area (acre), 

(9) . " lateral-surf ace (feet«*2) 

, (10) top-surface (feet**2): real,; ■ , • 

(11) begin 

(12) read (radius, height); '* - 

(13) * lateral T sur£ace := 2*PI*(radiiss/12)*height; ~ . ' 

(14) top-surface := PI« (radius/1 2) **2 1 
<15) area :=- lateral-surface + 2* top-surface; 
(16) volume := PI *((radius**3)*height).; 

• •(17) write (area, volume).; ' ' . • ' 
(18) end; , 

• » 

4.21.7. Effectiveness. The effectiveness of units checking is' limited* only 
by the capabilities -of the* units processor. 

Simple units checkers' may only be abTe to verify that -.two variables * are 
comunsurate., but not- determine if proper conversion factors have been applied. 
That isf a. relationship such as 12 inches = feet may not- be fully used in 
checking the computations 'in a statement, such as line 13 of the example. 
Ihere we asserted- that (radius/12) would be- interpreted as converting inches 
to feet. Ihe checker may hot support this kittd of analysis, however, to avoid 
ambiguities with expressions such as "one- twelfth of the radius." 

4.21.8. Applicability. Certain application areas, such as engineering and 
scientific, often deal • with, physical* units. In others, however, it may be 
difficult to find analogies to physical units. In particular, if a ; program 
deals only in one type .of quantity, such as dollars, the technique would not 
be useful. * : 

Units checking can be performed during all • stages of software development, 
beginning with requirements specifications. 4 
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4.21*9. LearningiUJj^jnehsional analysis is-* . commonly : taught in first year 
collie physics on statics; conversion frcm[ English to metric units is common 
throughput society. Direct application of these principles in programming, 
-xising . a ' units ; checker, should require no additional training" beyond 
understanding the capabilities" of the. specific units checker and the-means for* 
' specifying units-related information. - * - ■ ' . 

4.21.10. Cost. If the units checking capabilities are^incorporated directly 
in s compiler its usage "cost should be negligible.. If a preprocessor is used, 
such systems are : typically much slower than 3k compiler (perhaps operating at 
-1/10 .compil.ati.on .speed), but; only a single analysis of the program is 
required. .The analysis is only -repeated when the program is changed. * -• 
••*.•.*:.«• ■ ' ■ 

•;4.21,1U- References.' 

• . • . • ' . -. • . . - - c 

* * — . 
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• 4.22.1.. Name: Regression Testing ' - 

4.22.2. Basic features. Regression testing is a technique whereby spurious 
errors caused by system modifications or corrections may be detected. 

• - • . : ' . ■ " v • ■ % ■ 

4.22.3. Ihformatiori input. Regression testing requires that a set of gysteqi 
test cases be maintained aid availaBle throughout th4 entire life of the 
system. The test cases should be complete enough so that all of the system's 
functional capabilities are thoroughly tested. If available, acceptance tests 
should be used to form the base set of tests. ■ . > 

In* addition, to the individual test cases themselves, detailed descriptions or 
samples of the actual expected output produced by each test case must alsp be 
supplied .and maintained. 

4.22.4. Information 'output. f Thf output frcm regression te?sting'ig simply the 
output ^produced by\ the system frcm the execution of each of the- individual 
test cases. When the output frcm previous acceptance, tests has been kept, 
additional output frcm regression testing should be a ccmparisorTof the before 

. and after executions. ' ; - ^ 

4.22.5* OutlirJe of method,. Regression testing is the process of retesting 
the* system in : order, to detect errors -which "n^y have been "caused by program- 
changes. The technique requires ..the utilization of a set of test cases which 
have been developed (ideally, using functional testing) to test all of the 
system's functional capabilities. If an absolute det^raination of portions of 
the system which can potentially be affected by. a given change can be made, 
then only those portions heed to be tested. Associated with each test case- is 
a ■ description or sample of the correct output for that test case. When the 
tests have been executed, the actual output is compared with the expected 
output for correctness.* As errors are detected during the actual o|>eration of 
the system which were not detected by regression testing, a test ease " wh-Lch 
could have uncovered the error should be constructed and included with the 
existing test cases. ^ - j . • - - : . * • 

Although not required, tools can be used * to aid in 'perform*^*, regression 
testing. .Automatic test harnesses can he used to assist the managing of test 
cases and in controlling the test execution.- File comparators c5h often be 
useful in verifying actual output with exuectedwoutpljit. Assertion processors 
are also useful in verifying the correctness of the output for a given test. 

4.22.6/ Example* — . 

a. Application. - A transaction processing system contains a dynamic 
data field editor r which provides a variety of input/output fielg editing 
capabilities* Each transaction is comprised of data fields a§Jppecified by a 
data element dictionary entry. The input and output edit routine used by each 
data field is specified by a 1 fixed identifier contained in a data field 
descriptor in the dictionary entry. When a transaction ik input, each field 
is edited by .the appropriate input editor routine as \pecified in the 
dictionary lentry. Output editing consists* of utilizing output-editor routines 
.to format the output. 



b. Error. An input-edit routine to -^t^:nuneriev:data fields was'' 
modified ' to perform a fairly restrictive JRange cheek needed by a~ particular 
transaction program. Current sjrstein docfaiehtation indicated that - * this 
particular edit routine was only being "used by that single transaction, 
program. However, the documentation- was ^hbt up-to-date in that/^hother, 
highly critical, transaction program also used the routine, often with data~ 
falling putside of the range check needed by the. other program. \-> ■- 

c. Error discovery. , Regression testing would uncover the error giv«n 
that a sufficient _ set of"" functional' tests were used Cor ^performing the^ 
testing. If only the transaction program for which the modification was made 
were tested, the error would not have been v discovered until actual operation- 

. * - * * w " "■ ■".:*:?.- . 

4.22.7., -"Effectiveness. The eff^tivefcess of the technique depends upon the 
quality of the data used for performing^ the regression Resting. If functional 
testing, i.e. tests based on the functional requirements, is used to* create " 
the test data, the effectiveness is highly effective. The burd v en and expense 
associated with the technique, particularly for small changes,' can ^aj>pear to 
be prohibitive. It is, however r - w often quite straightfon^ard to determine 
which functions can be potentially affected by a given change. In such caseS, 
the extent of the testing can. be reduced to a more tractable size. > 

4.22.8. Applicability. This method is generally applicable. 

4.22.9. Learning. No special trainfrig is required ih order to apply . ; the 
technique. If tools are used in^-support of regression testing*, however, 
knowledge of their use will be required.- Moreover, successful application <5f 
the technique will require ' establishment of procedures and the management 
control necessary to ensure adherence to those procedures, 

4.22.10. Costs. Since testing^ is required v as 'a result of system 
modifications anyway, no additional burden'need result* .because of the method 
(assuming that only the necessary functional capabilities are retested). The 
use of tools; however, to support it could increase the. cost but i,t would also 
increase its effectiveness. - \ 
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4.23.1. Name. Requirements. Arialyzerv - 

.4.23.2. Ba^ic^ features. m The rjequiremerits for a systen will normally be 
specified using seme formal language which may be graphical and/df textual in 
nature. A* requirements analyzer can check for syntactical errors in the 
^requirements * specifications and ' then produce .a useful .analysis of the 
^relationship? between system inputs, outputs, processes, and data. Logical 
inconsistencies <3r ambiguities in the specifications *can also be identified'by 
the requirements -analyzer. . " ■ n . ■ . / i 

"4.23. 3* : Information input. The form and. content of the input will, vary 
greatly for different requirements languages. , Generally, there ^ill be 
requirements regarding what the system must produce' (outputs) and what types 
of inputs it. must accept. There will usually be specif icatipn§. describing the 
types of processes* or functions which. the system must apply td the. inputs in 
order, to produce the outputs. Additional requirements .may cbntern timing and 
volune of inputs, outputs, and 4 processes as well as. * performance measures 
regarding such: things as response time and reliability of operations. The 
form of the* inputs to the requirements analyzer 4s specified by the 
requirements specification language and varies considerably for different 
languages. Ifrv^cme cases all inputs are textual, -whereas seme languages' 

.utilise all graphical inputs -from a display terminal (e.gi.., boxes m£ght 
represent processes and arrows between boxes - might represent information 
flew). " \ , 

4.23.4. Information output. , Nearly ail analyzers .produce error reports 
showing * syntactical. . errors or inconsistencies in the specifications. For 
example, the -syntax, may Require that the outputs from a process at^one level 
of system decomposition must include all outputs from a decomposition of that 
process°at a more detailed level. Similarlyy^jor each system output there 
should be a* process which produces ttoat^output. *Any deviations from these 
rules, would result in error diagnostics. ^ - 4 v * 

Each requirements. analyzer produces a • representation of the s£s£em which 
indicates static relationships among system inputs, outputs, processes, and 
data.. Some analyzers also represent dynamic relationships and provide an 
analysis of ' them./ Thij/may ^ a precedence relationship, e.g.", process A must 
execute before process :B. It may; also include* information regarding how often 
a given process must execute; in order to produce ' the volume of output 
required. Some analyzers produce a detailed representation of relationships 
between different .data items. This output can sometimes be used for 
developing a data base for the system. A few requirements analyzers go even 
further and provide a mechanism for "simulating -the requirements using the 
generated system representation including the performance and timing 
requirements. - 

4.23.5. Outline of method. The user must provide the requirements 
specifications as input for. the analyzer. The analyzer carries out the 
analysis in an automated manner and provides it to the user who ^must then 
interpret the Results. • Often the user dan request selected types of outputs, 
e.g.^ an alphabetical list of all the processes or a. list of all the data 
items of a given type* . Some analyzers can be used either interactively or in 
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a batch mode.- Once the requirements specif ications- are* considered acceptable 1 
a few analyzers provide the capability for simulating the requirements. • It ik 
necessary that the data structure and data values generated' frcm the r 
requirements specif ications- be used as^inpufr to the simulation,, otherwise the ' 
simulation may not truly represent the requirements. < / 

4.23.6. Example. Suppose that « process called PROCESS B produces two files 
named H2 and H3 from an input - f ile , name, M2. (The purposes of the files are 
irrelevant to the discussion.) Suppose also that: PROCESS D/ accepts Files H2 
and H3 as^input and produces Files J3 and J6 output. In' addition, PROCESS G 
is a subprocess qf PROCESS D and it accepts Fi>e H3 as input.and produces File 
Jo. 4. Then tte.pseudo specification statements, figure 4-. 23; 6-1 , might be used - 
to describe the requirements.' (Note that, these requirements ar> , close to 
design, but this is often the case.) . ." 

PROCESS B' • 



USES FILE M2 
PRODUCES FILES H2, H3 

PROCESS D 



« 1 



USES FILES H2, H3 
PRODUCES PILES J3, J6 V 

• • *: 
PROCESS G 



SUBPROCESS OF PROCESS D 
USES FILE H3- ' 
PRODUCES FILE J6 



Figure 4.23.6-1 Requirements Specification Statements * 

The requirements specifications imply a certain .precedence of operations 
e.g., PROCESS D cannot execute until PROCESS B has produced files H2 and H3! 
Detailed descriptions of what each process does would'* normally be included' 
but are' emitted - for brevity. The . requirements analyzer would probably 
generate a diagnostic since the "statement for PROCESS D fails to indicate that 
it includes, the subprocess G. A diagnostic would also be generated unless 
there are other statements which specify that/file M2, needed by PROCESS B, is 
available as an existing file, or else' is produced by some other process. 
Similarly, other processes must be specif ied-which. use /files J3- and" J6- ' as 
input- unless' they are ■ specified as files ' to . ; be output frcm the system. 
Otherwise, additional diagnostics would, be generated.- It can be seen that 
some of the checks are similar to' data, flow analysis for a computer program. 
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^However, for l^ge §^t6ns;the analysis of requirements becomes very cdtaplex 
£if~ requirements J for timing 'and performance are included, and" if timing^d • 
•~ volme; analy sisjare to be carried out* . ( Volume,, analysis "is * c^ncernedjwittL 
sufch things as 1 how .often various! pr ocessed must execute if the system is to 
accept ^nd/or produce: a specified voltfae of data, in a single given period* of 
/time.) / y- ' V . • - 

4.23.7*. Effectiveness. Seme requirement -analyzers are very , effective for 
maintaining, accurate requirements .sp^ifications. ^ For large systems "with a 
-large nunberiof requirements they are essential . *Orv the other "hand, - most 
m existing, requirements arial^zefs are rather;, expensive to Obtain and use, and 
they may not be cost effective for development of small systems; 
■ ■■ ■- * ^ • - ... -.. .> 1/ . ' *. ; ■ v. >- ■ 

4.23.8". Applicability. Requirements analyzers;... are applicable, for use in 
developing ^ most . systems. They ^re^ particularly 'useful . for analysis of 
requirements^ or large anfl. complex Systems. - «y ~, 

4.23.9. Learning. Most requirements analyzers require a considerably amount 
-of training of persoir- 1 




? -V' " m 7 

4.23.10. Cost. Mostl30quirements analyzers are expensive to obtain -and use. 
They generally require a .large amount -<5f storage within a compote* and so can* 
only be used o^large computers. « m r 

4.23.11. References. ;■ ■ ~ 

' ■ V- " >■ V ■ ■ ■ '■■ * ' • ' ' 

(1) ALFORD, Mack W., "^Requirements Engineering Methodology* for Real- 
Time Prbcessing . Requirements, n TRW Softwares Series , TRW-SS-76-07, Systems 
Engineering and Integration Division, September 1976. * *. >^ 

" * . ■ 

* * (2) TEICHROEW, Daniel, n A Survey of Languages for Stating Requirements 
for CcmputernBased Information Systems," Th e University . rof Michigan, 
Proce e di ngs £fcg EaU Joi nt Ca n pvte r Confer ence , 1 972,pp> 1203-1224. * 
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. >.24.i: JIme. Requirements Tracing. v > ' • * v 7 : 

4*24.2. Basic 2 features. / Requirements tracing* pmvi^fea means of verifying, 
that tb£ • software of a system-addresses each r^uirement of that System ? and 
that th e t^tin^ software produces adequate ^and ^ appropriate responses 

to those requirements. v . \ v ^ y^Mit - v 

4. 24. 3 v Information input. The information needed to perform requirements 
• tracing consists of a set of : system' requirements and :'the software which 
embodies the bapability to satisfy the requirements. *• ^ \ x • 

4.24.4. Lif orma,tion output. . ' The information output tjy requirements tracers 
•is • the correspondence -found between the ? requirements of a system and/ the 
^Oft^are that is .intended to realize these requirements. - ■ 

4.24.5. Outline of method. Requirements tracing genially serves two major 
purposes. The first is to ensure that .each specified requirement of a System 
is addressed by ;ah identifiable element of the System software. The second is 
to, ^ensure- ttot the testing of that software produces results which are 

. ad^uate responses in satisfying each of these; requirements. 

. " : * i ■ . v ' ' ■ * v - A * ' '-it'?.'" ■• ■ "' : 

X eo^^ used to assist in making thesevlassuranceis^ is the ' use of 

ytest< ev£L^ these matrices, represent a /visual- scheme of 

; identifying wh^cir requirements of a . system have been Vctequately and 
appropriately addressed and ^ which have riot..; There are two bafeic forms of test 
ev&uatib^ iTirst forto: identifies snapping that -exists between. 

. the' requirement specifications Cf' a sys^ 

-'This, matrix determines whether each requirement is realized t?y seme module in 

. the System, and, conversely ,. v*iether each module is directly associated with a 
specific system requirement. If the matrix- reveals that a requirement is not 

^addressed* by any module, then that requirement has probably been .overlooked in 
the software design ' activity ■'. . If a modulfe . does not correspond to any 
requirement of the 'system, then that modulevis superfluous to the system. In 
either case, the. design of the software must be further and the^ 

JsjNstem must • be ; m6difi v ed_ accordingly^ to an ; acceptable 

requirements-design mapping. * - 

The serond^ora of ^ test evaluation matrix provides a similar mapping, except s 
the mgEpififf . exists between /the modules of a system and the* set of test cases 

; performed *6ri the system. This matrix determines which modules are invoked by 
each ttest case. Used with jtfae previous matrix,, it also determines which 
requirements will be demonstrated to be satisfied t>y the execution of a 
particular test case in the test ijlan. During actual code development, it can 
be usedvto . determine which "requirement specifications will relate to a 
particular module. In ; this way, if is possible to have each module print out 
a message during . execution of a test/ indicating, which requirement is' 
referenced ty the execution of this, module. s Ttie code mdSule itself may also 

M contain comments about the applicable requirements. ; 

, If these matrices are to be used* most effectively J.n a requirements .. tracing 
activity, * the two matrices should r*e used together. The second matrixes 
built prior to software development. Jkf ter jbh$ sof tarare has been \dey eloped' 
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• •. ; .* ,, : . •. ; /\ V' ' . ' • • , - • 

:: -:'tfe ^test ; 'c^;^-&;^d«l9tid' (based upon this matrix), it is 
necessaj7 ; to deternahe whether the execution of the test plan will actually 
• . demonstrate .satisfaction' of the requirements of the software system. By 
, .analyzing the results of. each test case, the first matrix can be constructed 
. :to vdetermine the relationship / that exists" between the requirements and 
softw a re re ali ty. . - — ■ ' . : < : ■ — - — . 

' .: • . . •• ' . y.y ■ • 

•• •- ' - • „•.'.' >"' ' *'• "' •■ 

The first matrix is mainly useful for analyzing' the. functional requirements of 
-a system. However, the second matrix.; is also useful in analyzing the 
_ performance, interface, and -design requirements of the system, in addition to 
the -functional requirements. Both are. often used in' support of a more general • 
requirements -tracing activity, that, of preliminary and critical ,design 
, reviews. This is a procedure used to ensure verification of the traceability 
tf/i 1 35076 mentioned requirements to _ the design of the system. In 
addition to the ^se; . of test evaluation matrices ,* these design reviews bay 
include the tracing of individual subdivisions .in the software design document 
back to applicable specifications made in the requirements document.; This is 
- f constructive technique . used to ensure verification of requirements 
traceability., ; '. v • ; 

• 4.24.6. Example. ' ' . •• ' ' ' ' '* - " 

a. Application. A new payroll system is to * be tested, \nong the 
requirements of this system is the specification that all 'employees' of a*e 65 
or older: .-:*•' ..•••'•'*••.'. s - • . - ~ ■ ^ 

/ 1.; receive semi-retirement Benefits, and ' ' ' . 

'2. have their social security tex>ate readjusted. 

To ensure that these particular requirements are i : appropriately addressed in ' 
the system software; test evaluation matrices have been constructed and filled • 
out for the system.' ' , 

b. Error.- An 'omission in the -software causes the social security tax 
rate of individuals -of age 65 or older to- remain unchanged. 

* . . < * .'. .• ■ ■ ■■ * • » ... ■ ' 

c. -' Error discovery. The test evaluation matrices reveal that- the ' 
requirement that employees of age 65. or older have their, social -security tax - 
rate- adjusted has not been addressed -by the payroll program. No module in the 
system had been designed to respond to this specification. The software is 
revised accordingly to accommodate this requirement, -and a test 'evaluation 
matrix is used to ensure that the added module is tested in the set of test- 
cases for the -system; . ' <..'"i" ' 

4.24.7. ^Effectiveness. Requirements tracing isv a^ highly effective - technique 
in discovering errors during the design 0 and coding phases of software 
development. This technique has proven- to be, a valuable aid in verifying - the 
completeness, consistency,. and testability-- of "software. ,If a system 
requirement is modified, it also provides^much assistance 1 .in retesting 
software by clearly indicating which modules must be rewritten and retested. 
' Requirements tracing can be a very effective 'tfechnfqug' in detecting errors 
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early in the software development-.cycle which could otherwise prove to be very 
.expensive if discovered later. • \ • y 

•••t^ifc TMs technique is generally applicable in large -or 
^ small system testing and for all tyfces of computing applications. However if 
-^2^? requir ? ent f th emselves are not cl early^if led anA g^Sted! 
ap?nLtS? irOTe ^ 8 ^ be . ver y'* di^cult to. accomplish in.-any 

• ?; 24,Sli .Learning.., Knowledge and a clear understanding of the requirements of 
<2LifJ S ^ 1S e ?f*j t }al. Mo"© complex systems will result in a corresponding 
. increase xn required learning. /--•.;" • ™ UAi * 

4.24.10. Costs. No special tools or equipment are needed to carry- out this 
oS!!-?^ 1 ^u d ? na ^"^iy- "^e major cost in requirements tracing is that 
• associaped with human labor expended. Requirements tracing is often I feaSre 
of requirements analyzers which are expensive to obtain and use. Ieature 



4.24.11. References. 



^^}V "IHREADS: Aj Functional Approach tbj'project Control. "Computer 
Sciences Corp. , EL Segundo, California Ji975. • _ ' " K ^ V7< 

M ^L 1 ? 12 ^ W * C * ' - nAn Experimental Analysis of Program Verification 

.Methods," EUL, Jbe§i|, University!* North Carolina, 1976.' 



.. >- V. v : Page 104 ' 

4*25. Name. Software monitors. 

c<^ ••• /*:\,' ; . ■ . * ' • • - >~ \'\ 

4*25.2. Basic features. These tcols monitor tte in 
order to locate and identify possible areas of inefficiency in the program. 
Execution data is obtained while the program executes in its normal 
. environment. /At the .end of execution, reports are generated by the monitor 
sunuarizingrrtte ~ ■ • " . ■ . : ~ ~ 

4. 25 .3 • 'Information input. Software monitors require as input the program 
source code to be executed and any data necessary for the program to run. 
Certain commands must also be provided by the user in specifying the 
information to be extracted by the monitor ^ahd in specifying the format of the 
generated output reports. Ihese ocraiands'may specify:; 

o what. is to be measured (e.g.-, execution times, I/O usage, core usage, 
paging actiyity,- program waits), - V "V 

o the specific modules to be monitored, _ : ■ 
;o the frequency that data i§ to be extracted during program execution 
' (sampling interval), f* ^ * 

> o the titles, headings, content of each output report, 
^o the units used to construct graphs,. 

^whether the gratis are to be displayed a& plots or histograms. 

i^.^\3^cn^tiA output. The output of a software monitor is a set of one 
.or more reports describing the execution characteristics of the program. 
Information that may be contained in these reports is given below. 

o A sunnary of all f the sample counts made during data extraction, : 

e.g. , the number of sanples taken where the program was executing . 
_ instructions, waiting for the completion of an I/O event, 'or otherwise 
bilked from execution. 

o A OTnnary of the activity of each load module. 

o An instruction location graph that gives the percentage of time spent 

for each^ group of instructions^partitioned in memory, 
o A program timeline that traces" the path of control through time, 
o A control passing sunnary that gives the number of times control is 

passed frcm one module to another, 
o A wait profile showing the number of waits encountered for each 

group of instructions. * ^ 

o A paging v actiyi1gr profile that displays pages-in, and pages-out for 

each group of instructions.^ 

This information is often .represented irT histograms and/or plotted graphs. 

4.25.5. Outline of method* ^ Software monitors ' typifcally consist of two 
processing units. The first unit runs the program being monitored and 
collects data concerning the execution characteristics of the program. The 
second unit reads the collected data and generates reports from it. 

A software monitor monitors a program -ty determining its status at periodic 
intervals. The period between samples is usually controlled through an 
elapsed interval timing facility of the operating system. Samples are taken 
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from . ^ entire address, range addressable by the executing task. Each sample 
may contain an indication of the status of the program, ' the load module in 
which^the activity was detected, and the absolute location of the instruction 
being executed. «5mall -sample intervals increase sampling accuracy but 'result 
in a corresponding increase iif the overhead required by the CPU. 

Thenstatistics •gathered~by :i£e~data extraction unit are • collected and 
summarized in -reports generated . by the data analysis unit. References to 
program locations in these reports will be in terms 'of absolute addresses." 
However, in .order- to relate the absolute locations to source statements in the ? 
program, the reports also provide a means to locate- in a compiler listing the 
source, statement >that corresponds to that instruction. In this way, sources 
of waits and program locations that use significant amounts of CPU time can be 
identified directly . in the source code; any performance improvements to the 
\progr am will occur at these -identified statements. - ' 

Software monitors are similar to another tool used to monitor, program 
execution, test coverage analyzers. ^TesV coverage analyzers keep track of and 
report on the number of times that certain elementary program constructs in. a 
program, have been traversed during a sequence of tests. During the monitoring 
of aNprogram^oth tools count, the frequency that certain events occur. After 
program exafition, both generate .reports summarizing 'the data collected. 
However i becSuse these tools serve different functions, ' they are different in 
their techniques of gathering information and in the type of information each 
collects. Test coverage analyzers are used to measure the completeness of a 
set of program tests, while software monitors measure the resource jdsage of a 
program as 3 means of evaluating program . efficiency." As an evaluation of - 
program, efficiency requires consideration of execution time expenditure, 
softwarelnonitors Utilize. ja strict timing ^mechanism during the collection of 
data. This is absent .in monitors such as test coverage analyzers which are 
not used to evaluate program^pefformance. . - ^ 

4.25.6. Example. \ .* ■ 

. . ..'.'« • 

a. Application. A program that solves a . set - of 1 simultaneous 
equations is constructed. The program first generates a set of coefficients 
and a righfc hand side for the system being solved. It" then proceeds to "Solve' 
the system and output the solution. 

b. Error. In 'the set of ca lculations required to solve the system, a 
row of coefficients "is dividedTJy-a^constant andHJien subtracted from another 
row of coefficients. The divisions are performed wi thin a nested DO-lopp but 
should, be moved' outside the" innermost . loop, as me^dividend- and- divisors ■ 
within the loop do not change. j. ' ' 

c. -Error discovery. The performance of the program^isr evaluated - 
through the use of a software monitor. Examination of the output reveals that 
the program spends almost 85% of its time in a particular address ^range. 
Further analysis shows 'that T6.65J . of all CPU time is used by a single 
instruction. A compiler listing of the program is. used to locate the sourcl 
statement that' generated this instruction, which is found to be. the statement 
containing the division instruction. Once the location of Jbhe" inefficiency is 
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discovered, it is left to the programmer to, determine whether and how the code / 
can be optimized.- . ' - ; , ,; / " 

4.25 .7 • Effectiveness. Software monitors are.vvaluable tools in identifying 
performance problems Yin a phograi. Their overall effectiveness, however, is 
dependent ppon the quality of their use. 

" ' .- * ■ . *■ 

4.25.8. Applicability.. ^Software monitors can be applied to any kind of 
program in any progranming language,. ■■ > m . 

4.25.9. Learning.^ There ar^ nbj special learning requirements for the use of 
software monitors* In order to use the tools effectively, however, the input 
parameters to the monitor must be carefully selected in determining the most 
relevant , reports to be generated. Once the. areas of a program which are most 
inefficient have been identified, it requires skill, to modify the program to, 
improve its performance. ^, 

4.25.10. Co6ts. The largest cost in using a software monitor is that . 
incurred by the CPU to extract the data during execution. In one 
implementation, extraction of data Resulted in an increase of user program CPU 
time by 1% to 50%. Storage requirements, also increase in order to provide v 
memory for .diagnostic tables and the necessary program modules of the tool. 

4.25.11. References. ' * 

(1) "Problem Program Evaliiator (PPE) User Guide, " Boole and Babbage, . 
^Inc, Sunnyvale, California, March, 1978. 
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* ^September, 1975. . . h 
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4,26^1 y^Name. Specif icatioiv-Based Ftmctional Testing. • V 

4.26 .2. Basic features. Functional" testing'can be used : to generate system 
test /data frcm- the information in requirements and design specifications. It 
is used, to test both the overall * functional capabilities of a system and 
functions which originate- during, system design. ' v ' ^ . 

"4.26.3. Information input. ~ ~~ ~; ' 77 v: ;' - • * .. 



■ •J , a. Data information. The • technique requires the availability of 
detailed- requirements and design specifications and, in particular, detailed 
, descriptions of input-data, files and data, bases. Both the concrete and 
algebraic abstract' properties of all data must be described.. Concrete 
properties include type, value ranges and bounds, record structures, and 
bounds on file data structure and data base dimensions. Abstract properties 
include subclasses of data, that correspond .to different ~ functional 
/capabilities in- the system and subcomponents of compound data items that 
/ correspond to separate subfunctional activities in the system. " 

/ b. Function information. The requirements and design specifications 

must also .describe the different functions implemented in the system. 

• • Requirements functions correspond to the overall functional capabilities of a 
. system or to subfunctions. which are visible at the requirements stage and' are 
necessary ito implement overall capabilities. Different ; overall " functional 
capabilities correspond to conceptually distinct classes of operations that 
can be carried- out using the system. Different kinds of 'subfunctions can also 
be identified. Process descriptions in structured specifications-, for 
example, describe data transformations which are visible "at requirements time 
and which correspond to requirements subfunctions. Requirements 'subfunctions 
v also occur .implicitly, in data base schemata. Data, base functions are used to 
reference, update and create data -bases and files. 

The designer of a sys£em- will have to . invent both general and 'detailed 
functional -constructs in order -to implement the functions in requirements 
specifications. Structured design techniques- are 'particularly useful for 
identifying and. documenting design functions. Designs are represented as. an 
abstract hierarchy of functions. The functions at the top of the hierarchy 
denote the overall functional capabilities of a program or system and may 
■correspond to requirements functions. Functions at lower levels correspond to 
the. .functional capabilities required to implement the higher level functions. .< 
General design -functions of ten correspond to modules or parts, of programs 
which are identified as separate functions by comets. Detailed design 
functions may be invented during the programming, staglMBfsystem development 
and may correspond to single linens of code. 




4.26.4. Information output. Jhe output to' be examinedl^fcds on the nature 
of the tested function. If it is a straight input^i Ht function, then 
■output, values are examined.. The testing of other classesTof functions may 
involve the examination of the, state of a data base or file. 
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•4.26.5. \ (Xrtline of method. -The basid - -idea , in j*uhctional . testing * is to 
identify, "functionally, important" classes of* data. The two' most important ' 
classes of dajfca are extremal values and special^ values. Different ' kinds of 
.sets of data have different kinds of^extremal values and different classes of 
"-'special values must be used to test different kinds of functions. -- 

' • , a. Ex t remal values. The simplest kinds .of. extremal values are 

rociated with elementary data items^Vff a variable is constrained to take 
values which lie in the range (a,br, then the extremal values are a and b. , 
a variable is constrained to . take on values frcm;a small set of discrete 
values then each of those values can be thought of as "an extremal case'. 

The construction of extremal' cases for .data structures (e.g., group data 
items) can be more complicated. It is necessary to construct extremal* values 
' of both -the component elementary parts of the data structure as well as its 
dimensions. The data structure can be treated as a single quantity. In this 
case, when it takes on an extremal value all of its elements take on that 
value. It is also possible to consider its components as a set of values in 

.which one, more* or -all of. the . components have extremal values. The 
construction of extremal values for files and data bases is similar to that 
for data structures. Files with extremal ^dimensions contain the smallest 
possible and largest possible number of -records. If the records are variable 

. sized they contain records of the smallest and largest dimensions. .-j 

b. » Special values. There^ appear to be two kinds of special ' values 
that are. important for data processing programs. The first is useful for 
testing functional capabilities in which data is moved* around from one 
location to another, as in a transaction-update program. Functions of this 
type should be tested over distinct-sets of data (i.e., values in different 
•> files, records, variables or data structure elements should be different) in 
order to detect the transfer of the incorrect data frcm.the- wrong- 'soiree or 
into the wrong destination. The second kind of special data is useful for 
testing logical functional capabilities that carry out different- operations on 
the basis of relationships between different, data items. It is important to 
test functional capabilities of this, type over special values such as those in 
which sets 'of data that enter into the 'comparison are all the same. - 

Additional kinds of special values are important for scientific programs or 
programs which do! arithmetic calculations. They - include zero, positive and 
negative values "close 11 to zero, and large negative and positive values. 

Functional testing requires that tests be constructed in which the input' : data 
is extremal, non-extremal and special as well as tests that result in program 
output that is extremal, non-extremal or special. 

4.26.6. Examples, f ' * . 

■•' ■ ■ ■ ■' ' • : • - '■ :'- \ . 

Example 3: "Testing of requirements functions. 
• • ••r2< : v"'" 'C- v. • ••' '. •.••'•»" ) ' ' ' ; ' ' ' •.'••••.'•':'■.-.: ... 

". • a. . ^Application. '.: A computerized dating, system was built in which a 
sequential file; of potential dates was maintained.' "Each client for the- 
".service .offered would submit a complete questionnaire' which was used to find . 
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the five • most; compatible dates. , Certain criteria had to be satisfied before v 
any potential data vras selected -and it is possible that no date could be found 
for;a clientsor less, than five dates found'. '*'-'■ " - , - > • - 

■■■ ' \* ^§* ror# in the faieL processing logic causes the program 

■ i ^f^V^f la ? fc Potential date in tee sequential file whenever there is^nb 
potential datt for a" client'.' "*** >-': . • 



" ^ijordiscovery. The number of dates which are found for- each 

S Ae «i- 'i S a diln r^s n - of tne output data and has extremal values 0- and 5. If 
the. -^find-a^ date" functional capability of the system is tested over data for 

v a client fc£. which no-date should exist <then the presence of the error will be' 

-...-revealed.; - '• • .' 

' ' Example 2i Testing of detailed design functions. 

.' ai Application. The designer of the ccmpuEeriBec<* dating system in 
Example 1 decided to process the file of potential dates for a '-client by 
reading in tee records in sets of 50 Records each. A simple - function was 
designed to compute the number of record subsets. 

bi Erro?. The number of subsets function returns the value 2 when 
there are less than 50 record? in the file.. - \ 

' ■ ■ ' c. Error discovery. The error will be discovered ifr the design 
function is tested oyer, the extremal case for which is should generate the 
minimal output value 1. Note that this, error ■ is .not revealed (except ' by 
chance)' when the program is tested at the requirements specifications level, 
it will also not necessarily be revealed unless the code implementing ' the 
design function is tested independently and not in combination with the rest * 
of .the system. ...... V % ,' : 

- 4.26.7. .'Effectiveness. Studies have been carried ' out which ' indicate 
functional testing to be highly effective;- - Its Use- depends on specif ic : 
descriptions of system input and output data and a complete list, of all 
•functional capabilities^ The method . is . essentially manual and somewhat 
.informal. If a formal language could be designed for describing all input and- 
.output -data /sets, then a topi could be used to check the completeness of these 
descriptions. Automated, generation of -extremal, non-extremal and special 
cases might be difficult since no rigorous procedure has been developed for 

: this purpose; . . •'■;.'•'. ■ : ' '....•:.. ■•■ :\ 

For many errors it is necessary, to -consider combinations of extremal, non- 
extremal and special vaiues* for "functionally: related" input data variables 
In order to avoid combinatorial explosions, combinations must be restricted to 
a small number of variables. Attempts have been made to identify important 
combinations (see references) but there are no absolute rules, only 
suggestions and guidelines. , • 

4.26.8. Applicability. Tttfg method is generally applicable* ^ 
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4.26.9. Learning. It is necessa^ 'to^ sane , expertise with the 

identification of extremal and special cases and to avoid the combinatorial 
explosions that may occur when combinations of extremal and special values for 
different data litems are considered.. It is also necessary to become skilled 
' in the identification of specifications functions althbugh this process is 
simplified if a systematic approach is followed for the representation of 
requirements and design. . 



4.26.10. Costs. The method^ requires no special tools or equipment and 
contains no hidden excessive tests. 

4.26.11. References. ; > / 
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4.2/. 1.. Name* Symbolic execution. * ... 

4,27.2. Basic features. Symbolic execution is 'applied to paths thnpugh 
programs.^ It can ,. be used - to generate expressions which describe the 

' c ? DUla £ ve effect of the computations .which occur in a program path. It can 
also be used to generate a system of predicates describing the subset of the 
input domain which causes a specified path to be traversed. The user is 

-^qp^^-^te^erlfy-^he c o rr e ctne^of^e-^Q tput which is generated"^ 
symbolic^execution in the same way that output- is verified which has been 
generated^ by executing a program over actual values. It is used as a basis 
tor data flow analysis and proof of correctness. • , 



4.27.3. Information 



r 




a. Source eckie. The method- requires the availability of the program 
source code. r~ • 

v J 3 : Pr °8 rani Path's. Ihe" path or paths through "the program'.: which -are to 
be symbolically*' evaluated . must • be specified. The paths may be specified 
directly by the user or,, in some symbolic evaluation systems, selected 
automatically. . 'v. ... . , . 

V ' ' c * 'Tfopab values. Symbolic values must be assigned to' each - of the 
"lnbut" -variables for the path or . paths which are to be symbolically 
evaluated. The user may ,be responsible for selecting these values or the 
symbolic evaluation system which is used may select. them automatically. 

. 4.27-4. Information output. • .-V ' 

V~ ■ ■ a - Values of variables. The variables whose final symbolic . values 
arey of. interest must-, be specified. Symbolic, execution will restilt in the 
generatiqn of -expressions which describe the values -^^. these variables in" 
terms of the dummy symbolic values 1 assigned to : input variables^ 

b. . System of predicates. Each of the braW-predicates "which occur 
along • a , programj path constrains the input: which causes that, path to' be 
followed-. The. symbolically evaluated system of predicates for a* 'path 
describes^the subset of the input domain that causes that path tp- be-followed. 

^.27.5. -Outline of method, * . .'•'." .-./-': - ' ' '•*•'•'*: 

- , a* Symbolic execution. Symbolic values are symbols standing. for sets 
of values., rather than- actual -.values. The symbolic execution Of a path is 
carried out. by symbolically executing the sequence of assignment statements 
occurring, in the - path,; Assignment statements are symbolically executed by 
symbolically.- evaluating the_ expressions - ion the right hand -side of the 
assignment., 4 The . resulting, symbolic value becomes the hew symbolic Value of 
the variable oh the left hand side. An arithmetic or logical, expression is 
symbolically, executed by substituting the symbolic values Of the variables in 
the expression for the variables, \ .' . .. ' 
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The branch conditions or branch predicates .which occur in •conditional < 
branching statements can be symbolically executed to form symbolic predicates. 
The symbolic system of predicates -for V a path *can be . constructed ' by - 
symbolically executing* both assignment statements and bnanch predicates during 
the symbolic" execution of- the., path;" The symbolic system of predicates 
consists of the sequences of symbolic predicates that are generated by the 
execution of the branch predicates. ■ '■ 
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b. Symbolic execution systems. .All symbolic execution systems must 
contain facilities for: selecting ''program paths to be symbolically executed, 
, symbolically executing paths, and generating the required symbolic output. 

Three types, of path selection techniques have been used: interactive, static 
and automatic. In the. interactive approach, the symbolic is 
constructed so that control returns" to the user each time it is necessary to 
^ make a decision as to which branch to take during the symbolic execution of m\ 
program. In the static approach, the user specifies the paths he wants 
executed in advance. In the automatic approach, the symbolic execution system 
attempts to execute all those program -paths having consistent symbolic system 
of predicates. , A system of predicates is consistent if it has a solution. 

The details of symbolic execution algorithms in different systems are largely 
technical. Symbolic execution systems may differ in other than technical' " 
details in the types of symbolic output .they generate.- Some systems contain, 
for • example;,: facilities for solving systems of branch predicates. Such 
systems are capable, of automatically generating test .data for selected program 
paths (i.e., program- input data which will. -cause the path to be followed when 
the program is executed over that data). 1 ' 

4.27.6. Example. '••» ' /'■■•• ' ' *' ; ■'" 

a. Application, A FORTRAN program called SIN Jras written to compute 
toe sine function using the McLaurin^series. . " ::• , 

PREDICATES: ""' " '4-' - ■. • •. ! ..' 

tt«*3^6).GE.£ w '•*/"' .^V c '< ' 

(X««5/120).GE.E " • • -'. :. ' .'•' • ■ ' ^ ."' 

(X««7/5040).X.T.E " ' ' - t " vi' 

output, ; '•••:••;■•',••'•;'. . ' " • y' :>','; 

. sin = ?sum r cx«*3/6) - (x**5/i2br ; <•' > ' V 

Symbolic output for SIN .••-•,«•"-" >^ ; : . ^ . * j 

Figure 4.27.6-1 .Symbolic Execution Example" v 

.. Errors. _ . The> program . .contained three errors, including an . 

■uninitialized variable, the use . :of ' the expression- -T»*(i?a) . instead of 

,(-1)**(I/2), and the failure to add the last^term computed 'in fhe series on to " 
the final computed sum. '• ■v-v".'' .&fV..% : 

■ v -:'.;>' -'.'\; ; / : ; ; ; : :;'' •• .'■•/'v.'-Vv;.- '"" v -.. v : • -,«.^ ' '•• 
:•. v . • '••:•,•/'-;•■/■ : .v ••,*■> •*••>:■ > . -: , 
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, Deferent >aths through SIN, correspond to different, -nunbers- of . iterations ypt : 
the loop in the program that is used to compute terms In the seaies. The' v ; 
symbolic output in figure 4.27.6-1 was generated by symbolically evaluating 
the path that involves exactly three iterations of the loop. 

c. Error discovery. . The errors in the program are -discovered by 
com paring the gyrcbplic " output with -the standard formula for the McLaurin ; 
- 6fcrlt». The symbolic evaLuator t nat was used to gt>r\t>r?t* the output^" 
represents .the values of variables that have been uninitialized with a 
question mark and the name ' . of the variable.: • Ihe error, involving the 
expression £-1)«*(I/2) results in the generation of the same rather than, 
alternating isigns in the series sun. The failure to use the last computed' 
term can/be .detected by comparing*; the predicates -for the symbolically 
evaluated^ path with the symbolic output value for SIN. •- 7 «• . 

4.27 .7< Effectiveness.; Studies have beep carried out which indicate that 
symbolic evaluation is useful, for discovering a variety of errors but that, " 
except in a small number of cases, it is not more effective than the combined 
use of other -methods such as dynamic and static analysis (1). 
J 4 ■ Vs.' ■ . 

^ One of the primary uses of -symbolic evaluation is in raising the confidence 
level, of a user in a program. Correct symbolic output expressions confirm to 
the user that the code carries out the desired computations. It is especially 

. useful -for nonprpgrammer users. 

•'• r • - . : . - ; .• ... ^''"V- 

,4.27 .8. Applicability. . The method is primariLy useful for programs written 
■ : in , languages which "involve operations that can be represented in a concise 
formal way. Most of the symbolic evaluation systems that have been built are . 
for _ use: with algebraic programming languages such as FORTRAN and PL-1. 
Algebraic "programs involve computations that can be easily represented using - 
arithmetic .expressions. It .is difficult to generate symbolic output from V' 
programs which involve complex operations with "wordy" representations such as 
the REPLACE and. MOVE CORRESPONDING operations in COBOL. 

4.27.9. \ Learning. It takes a certain amount' of practice to choose paths "arfe 
parts of paths for symbolic evaluation. The user must avoid" the selection of 
long paths or parts of paths that result in. the generation of expressions that 
are so. large - that they are unreadable. If the>symbolic evaluation system 
'being used. gives the user control over the types of expression simplification - 
that are carried out, then, he must learn to use this in a way that results in 7 
the generation of - the most revealing expressions. • ' V" J 

4.27.10. Costs. Storage and execution time : costs for symbolic evaluation 
hayie.. been calculated in terms -of program size, path length, number of program 1 
variables and >the* cost .of interpreting (rather than compiling and executing) a* 
program path. *' . * ■ . - . • • - v ^ _ . -fr'r \ .* 

v. ■ <• ■*•.."■*■. ■ *■ : •* 

The storage required for symbolically, evaluating a path of length . P in a 
program with S ' statements containing N variables is estimated to": be on the 
order of 10CP+S+V) <2)< Let C1 be the cost of preprocessing a program for 
interpretation, C2 the cost of inten>reting a program path, Cons is the: cost ' 
of checking the consistency . (i.e., solvability) of a . system , of .symbolic' 
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^ ^S v^^ «re; expressed in- unit& . 
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4.28.1 # : Name; Ite^t,. coverage analyzers. 

7 C28^T T;r 3aj5ic features. Test coverage : analyzers monitor the execution of a 
program - ^during program toting io^ 

of program 'tests, (^ple In terms : of the branches, 

statements or jother elementary 'program constructs wh^ch are usjed during the 
executidh of the program over the tests. V 

4.28.3 i Information inputs - Test coyetrage^ the program source 

coie and a set of program tests to geheriate test qpvSrage rej>orts« 
Sophisticated coverage analyzers may also involve input" parameters v that 
describe which of . several alternative coverage measures are to be used. 

4.28.4. v Information output. lypidal output consists of a report;, jwhich 
describes the relevant feature of the program which has been "exercised" over 
a sequence of tests. Branch coverage .analyzers keep track of and report on 
the number of times that each branch in a program has been traversed during a 

-sequence of tests fcl) • A program branchy is any; transfer of control from one 
program statement to another, neither thrdugh 'Execution of a control transfer- 
instruction* or ttrough normal sequential flow :of control from one statement to 
the next.- . /.■;,> -," J ■ . 

Different kinds of coverage analyzers will # report different kinds of 
information. Analyzers which .meSsure coverage in terms of pairs of branches, 
loop iteration patterns or elementary program functions have been proposed but 
branch coverage analyzers are the most widely used. In ^ addition to coverage 
information, -analyzers may aiso recbrd and print variable range and subroutine 
call information. The minimum and maximum values assumed by "each variable in 
a program, the minimupi and maximum nunber of times that loops are iterated 
during the -executions of a loop, and a record of each subroutine; call may be 
reported. , •>:-• •■■ \ ^ *' - 

4.28.5. Outline of ifi^thod. -\> '.'.V".f- ; 

a. Branch analyzers . Branch coverage analyzers typically consist of 
two parts, a preprocessor and a postprocessor. The preprocessor^ 
Hprobes" into the. program f3r which test coverage analysis is required. . 

The probes call subroutines or update matrices that record the execution of 
the part ot the program containing the probe. Theoretical studies have been 
carried out; to determine the minimum number of probes required to determine 
which - branches are executed during a program execution. The probes may also 
record information for determining minimal -and maximal^ variable values, loop 
iteration counts* and subroutine calls! v : 

The information which is generated by program probes, has , to.-be. processed 
before test coverage . reports ; can he generated * If a, sequent has 
been carried out; the inf onnation f rem the different tests has to be merged; 
The* processing > of the inf orm^tion generated fcy ^ probes -during , progr^ testing 
is processed and reports are generated -by the coverage analyzer postprocessor; 
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: b > Ftmctlbn analyzers: Function analyzers are based on the idea that 
, eacfr progra m . construct inplements r one or raye elementary functions. Loop 
constructs, Tor example, involve ftmctions which ^etermine^ if a loop is to' be 
entered, wh^ it is to be exited| how mar* times it is to W iterate, the. 
initial value of the loop index variable (if present) andvSubsequenfc;Values of 
the loop index; "It ;is. pwsible to define complete seta of tests Jbr these 
. ftu^tions whic*^ on at least one^ 

m r . test if the function contains; one of a predefined set of possible functional 
errors (2) • Test coverage analVzers can be built which keep track of *the data 
; 6ver which T instructs are execjuted and which report on the functional ; 
cdmpleteness of the data used in the execution of tttf : instructs. Function 
coverage : analyzers can be instructed using the preprocessor probe" insertion 
k and j»s^ generation approach uSed : for branch coverage 

-analyzers, f , ^ , T . .v • -y- ; % 

APPligatipni -A quicksort program was constructed which! cbntaihs a 
branch to a separate part of the program code . that c^ out an insertion * 
sort. The quicksort part of the -code brandies to: the insertion sort. The 
quicksort part of the cbde branches ]to the insertion sort, wheneyer the size ofr 
the original list to be sort^^ list - is below 1 

some threshold value. Insertion^ 

small lists' and sectiOTS of :iiste i>e^ in their 

execution time formulae. \VV v-.' • ' " ^ 

error, toe branch to the insertion. Bort is made 

original list, car the section of the list currently being processed, is* less 
than or equals to one. ' V > *' ^- : . ■< - 

c. Error discovery. 'Parts of the insertion sort code are not 
executed unless the list or list section being so&ed is: of length greater 
.than one. Examination of the output from a branch coverage analyzer will 
reveal that parts of) the program are never executed, regardless, of the program 
tests whifh are useO. This will alert arid draw the attention of the 

programmer to the presence of the error. v ; • 

. . ... ... ..... . _ ... ...... - :<;. . . •; :•; f, 

0 It is. interesting to note that ts error is not discoverable by the examination . 
of test output data alone since the program will still correctly sort lists. 

4.28.7. Effectiveness. Research results confirm that test-coverage analyzers * 
are a necessary and important tool for software validation^ Previously, 
assumed "complete"- test sets for production software have been found to test 
less than 50% of the branches in a program (1). -The use of test' coverage. v 
analyzers reveals the inadequacy of -such test sets. - 

Studies indicate' that although test coverage of all. parte of ;.-a- program is 
important, it is not enough to simply test all branches, or even all program 
paths. A large percentage of errors are only detectable when , a program, is . . 
tested over extremal -cases of special 1 values that are closely related to the 
functions performed. in the program. 'There appear to be 'three situations in- 
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yTwhich v branch coverage is effective in finding errors. The first is that in 
. which an error in part of a^ program is. so destructive that any ' test that 
— ^causes' that part o f the, program, to— he-executed -will result in incorrect 
output. The second is that- in which, parts of « program are never used during 
any . program execution/ and the \third that in Which unexpectetf parte .of a 
J program ane used during seme test. Other kinds of errors require ^additional 
test selection techniques, jsuch as functional testing. ':■* 

. . ' . ■ ' 'v :. • '. ■ ■ " \ " " o. 

4.28.8. Applicability. Test coverage 'analysis can be applied to any kind of 
/program in .any programming language. . , • 

4.28.9. Learning There are no special learning requirements for the use" ^'of 
test coverage analyzers. ' Once a set of tests has been found to be inadequate; 
it requires" skill to. generate data that will cause the unexercised features of 
.the program to be used during program execution. *, , 

:-^'~'A^30'.'i^.V^^t&Mi%^^- "^BBf^. analyzers can be' - inexpensive to use. The 

major ■ expense te the capital cost for" the tool I It is estimated that the" 
S^tructioh of :;a' i test coverage tool requires a level of effort which is more 
than that Ii*equii^l for a parser but less than twice that effort." " The major 

, part of test cover^e analyzer consists of the parser; 'that is used to 
.determine.' probe insertion points for a -programi . • 
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data^to exercise.a target program. They may" generate data through analysis of- 
tfce program itself or through analysis of the expected input to the program in 
its normal operating environment. Test data generators may use numerical 
integrators and random number generators to create the date. 

4.29.3. Information input. Test data generators require as input: 

a. the program for which data is to be ; generated, or * 

b. -a quantifiable .description' of the cfcmain of possible inputs to 
■ «. the program from which, the test data generator is to produce 

representative values. 
■ -. ■■ .. ■• • ■>■.) »•••.•';-. • - . . " • • ; \ 

\4.29.4. Information output. The output produced by test data generators is a * 
set of data that can be used effectively to detect execution- time errors in a 
program. ^ it is generally intended that such test data cause the program to be 

: . ^xer^is^ . when^executed. - It is, also- desirable to have this input " 

data be representative of the actual data used in real program operation in ' 
order to properly evaluate results tobtained from program" execution. 

4.29.5* Outline of method. Test "data- ^generators generate test data" for a 

.program m a /systematic, deterministic manner. There are- two major methods ^ 
.Wr^^ ;;can - be " implemented as 

xuAiy automated tools. ' •• 

One method of test data generation -analyzes .the structure cf a program and', 
based upon this analysis, ■ generates a set of test data wMc* will drive 
execution along a comprehensive- set of program paths. This method attempts to 
maximize ; the structural coverage achieved during execution with thederived ' 
flata. .^Biough this approach requires a- detailed, rigorous structural analysis 
a program, (which is often quite difficult, if not impossible), tools have 
-been developed which aid in the automation of this analysis. There are tools' . 
which- can^ analyze a program, and identify certain structural elements in that 
program. Data is then automatically generated that will drive execution 
through eachjjf these program elements. 

If i£ is desirable to increase the coverage achieved by the test data, - there * 
-5f* e^st -tools which use automated program.analysis to aid in accomplishing . 
this. After monitoring program-execution with the generated data, it may be 
possible to. increase the current ..structural coverage achieved- ^y using 
automated tools which assist-in determining how to alter the,- current 5'et of* 
test^data as necessary to cause different branching- conditions to occur* test 
data, generators that create -test data based upon - the amount of- structural 
coverage .that the data will; achieve. are-generally very sophisticated 'tools. 
Much research and development work is currently being done in this area. 

A- second approach to generating test data .•; is : -based upon analysis i of the • 
possible inputs, to , a progr^ real, operational usage.- This technique - '*< 

requires more knowledge of the software for which input data is to be * "-' 
generated than, the previous technique. However, in 'thfe- approach' the output 
generated from program ^execution provides , more meaningf44 'resulte to the user ■ N 
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during testing. One such tool that ..utilizes this technique examines the 
domain of all possible input values to! a program under normal 'program 
.operation and partitions this domain into" mutually exclusive subdomafns T F or 
each 'subdomain there is" an s as§oeiated probability that a sequence of actual 
input values., will belong- , to that partition. Data is then generated by 
sampling from- each subdomain with the distribution of sampling determined by' 
the ^subdomain* s associated probability. Automated tools have been built to 
assist in computing these probabilities and in "sampling from the appropriate 
partitions* '■ ' 

-Ms technique attempts to mirror the Intended - operation of V program : by 
generating test data which is representative :of its operational input. This 
mode of program testing <an be very, useful during a preliminary period " of 
software operational , use* Using this technique, , reasonably accurate 
predictions^ can made on the software 1 s performance In^jeal operation. : V 

" Other test data generators cacist which use liess sophisticated techniques than 
j those described above. Many of them generate date based upon commands given 
r by the user and/or from date descriptions in a program, such as in a COBOL 
^program's date. definition section. This is mainly a COBOL oriented technique 
in which the test date "is intended to simulate transaction inputs in a 
database management siteation. This technique, however, can be adapted to- 
other envin>nnents.;v ; \p '" :£■.. i - . - .■■ 
V r 1 V' \tS , /V V.' • ; ■ . */ : -;^ T — -- ---- 

4.29.6* E^mpl'e. Test date is required for a new payroll program. A test 
datia > generator is used to r ; generate data normally contained in the payroll" 
records of each employed on the payroll. She data ^fields in these records 
consists of: f.' v , . v • • * 

• orE^loyee .identification numb^*, ; ' 
*H K o' Employee name ', " & .. % ■■»•». , . . . ' . 

• o i^icatioti; jof hourly or Varied employee - 

- l > b Salary, rate ( if salaried) t " . - " - - . ' 

; 0 'Hourly rate (if hourly). * v . . , . ■ 

o Number 'ot incurs wrked during last ^ '. . " 

o flunber of tax ixemptions. declared . ". ' 

- o Federal withholding -tax. rate , / ' • •< • " " ' 

p Social security tax rate . • : 

o Marital status * '< . .. f . '"• ■ ■ ... ,* : 

A file of records containing this information is created by -'th% test date 
■ . generator; v For each- field in -a record, a 3 value with the appropriate data type 
is randomly generated (e.g., alphanumeric for Employee- Name,, integer for 
Employee Identification Number, real for Federal Withholding Tax Rate). The 
file i,s then reformatted' in- an organization that is acceptable to the payroll 
system as input." -The generated test data will then be fed to the payroll 

- program to be tested. ../„.. ■ .. ' 

4.29.7. V Effectiveness. The overall effectiveness of automated test, "data 
generators in use today is generally poor; Though these tools' permit the... 
generation of more test data than any » human tester could create., {thereby 
. devising more, test cases) ,'' a burden is created on the hUman^tester to' evaluate v 
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Si ^ tert-^results obtained from program, execution with the generated data; • 
• »™«*tonately, test data generators: themselves do not have a facility by which 
£ V *^J*^^ of-^the-test -data- generators^ 

date in v a 'manner which is totally insensitive to the H 
^ f 60 " 1131,1 " 63 of .a program.' The date may often be -meaningless in . 
content.^ It may^ focus testing upon, an unimportant portion of the program and j 
.^te^ ignore critical portions. : A human tester; nowever, of terf has a 1 

• certain intuition about which program areas need - to be more /thoroughly tested 

and' so ^creates his test date accordingly. The bVeraOLl ignorance 
2L2fV ^generators in determining which data items woulS^offer the most 
. -, Potential in discovering errors is the major factor behind . their current 
ineffectiveness in program testing.. '-•*- - 

-v? ":*>-■■..•_ . ... .... •■ r : ; ' ' V. 

Applicability. Test date generators are generally applicable for any 
system requiring input -date for operation. " < w 

KvJ' 29 ' 9 ' Learnings For those test data generators which only require as input 
tee source program for which test date is desired,' very little learning is 
• rB&rrt^uaz.t^u^-. The user.interface- With the tool SL alwSs bl 

• w«Sfji ^M^. for ' the tool should provide sufficient - 
formation Tor its operation. Jpr those date generators ; Which create date 
•based upon the domain of expected inputs to the program, much more learning is 

environment and operational- usage 4 of the software so that representative input 
date can be generated; - •.. :% .: AV .,r. .-' . v. - 

4.29.10. Costs. . Automated" test " date * generators are ' generally quite 
ex^jsiye. This, is primarily due to the relatively infrequent* use of these 
tools in actual- testing environments. The initial costs in building !fcest date 
ff^J ft W rareLy been offset by benefits obtained in usihg them.' 
t . :.. As yet, the d^ved ^Uizatibn of the more sophisticated tools thit exist 

• ^^jjSff? 6 ^ generatorf are among 
. tee most costly testing tools that exist today.' ^ . r 6 
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site used to- t^ •of software. "This test site sipwlates the 

enviroinent loixler :the j^tware will- normally operate, A test bed 
permits processing 
of intermediate outputs without - destroying simulated execution ^time, and 
' allows - full test repeata|ELity \arid diagnostics. To be effective, the 
controlled circumstanced of the test bed must truly . represent the behavior of 
the system of which the $dffaw 

4.30.3. Information input, " Th^iiif ormatipn input * a: jjtest bed vis the 
software, for ' - which ; a testing environment is to be jsiimalarod and Which will 
later be installed in a real system.;. \ r ^ v ; y *v-^ : 5V - 

4.30.4/ Jhfoj^tion outlet. The information output by a i£st bed are: the 
results obsert&i throu^ in the test bed^ 

JEhis inf ormatic h is used as a; preliminary means of determining Aether - the 
software will operate as intended - in its real environment. 

V- • ' ^'■■■^/■^ ,\ ., '.^ 

4.30.5. Outline- of method. Test beds provide an environment ih T' which to 
nunitor the operation of software prior* to installation in a real system.. To 
be of value, this* environment must realistically^ reflect those ~ properties of 
the system which will defect or be affected by the operation of- the .sdptware. 
However, the test bed should simulate only those ( * components - in the- system 
which the softwares requires as a m jniirran ^interface with the system. -^JJ^-'uSll'. 
permit testing to focus only on the software component for 'which toe test bed 
is built. v"-"'"''" ^ ■ :; .'- ;; ''v " V ' : :^" : -V"f : \"V 

Test beds are built through the considera1£oi> of, and proper balance between , 
three major factors: *, 1 : " ■ . " 

o the anjoiant or^ealism required by the test bed to properly reflect the 

operation of system- properties, r 
o resounds available to bi^ : '" ^ y 

o the ability: of- thetest b^ w focus* bniy on 1 the • isofta^e- being tested. 

Test beds come in many forms, depending on the level of testing desired. For 
.single module testing, a test bed insist merely of test data and a test 
driver. A" test driver is a program 1 i^di ^eds input data to the program 
module being tested, causes the module to be executed, and collects the output 
generated during > the. program execution. - J^a completed^ 'but non^final version, 
of settle to be tested, the -t£s£- ted may also include stubs. ' A stub isV. 
dunny routine that simulates the operation of a module that is invoked- within 
a test. Stubs can be as simple as routines that automatically return on a 
call, or they" can be, more • and return simulated; results. The final 

ve^sibiu \bf the^ ^ othe£ ^ 

larger total gystem. The test bed f or in the system may consist 

of jthose system components which directly interface with the component being 
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As mustrated in the above examples, . test 'beds ^permit the' testing of ~ 
component of a ..system without requiring the availability of the full, complete 
system. ' They merely supply 'the- inputs required by the soft ware component to 

^"Jfc executed, and provide a repository for. outputs- to be placed for "analysis; 
In addition, test beds, may contain monitoring - devices which > collect and 
display intermediate ; outputs during program : executibn. In this way, • test beds 
provide the means of >. observing the. operation of software as a component of a 
system without requiring." the availability of other system components, which 

' may be unreliable. ■ . . •. . . '• • . . • • , . J \ • 

4.30.6. Example. The federal government 'has just distributed to ail American ' 
corporations new tax rates to be imposed on the earnings of all employees 
beginning at the start of next year. Due to these new tax rates, Company XXZ 
•has- .had to revise its current payroll program so that it will accommodate the 
• n« . federal, regulations by January 1. 

In order to test, this .new programga test bed is being constructed to simulate 
?L°£?T atton • of payroll system. 'To simulate the inputs to this system, a 
test file, of data containing all the- information necessary for -the system to 
operate is created. - The me.consists^of a. record of information for each 
employee in the company. Each record contains the following ,data : 

o Employee ident4fi*ation number / ' ; . ' • -/,.•$%.•■ 

o Employee, name • ■ ~- : ; ■ ■ 

o -Indication of hourly or salaried employee 
* ' o Salary rate (if, salaried). .... ' y ■■- 

:■ '■' d Hourly rate <if/houriy)^: :; ,) ' ■ ; ' ' 

- o Number of .hours worked during last pay period - * 
o .Number of tax; exemptions declared / 4 :; 

o Federal withholding tax rate - V , 

— o Social security tax rate V-":* '. "■ • ; A ; ~ 

\ • o Marital status 

A test driver controls the executioh-of the payroll program. .. It feeds the ■ 
above;. data to- the program, in -the proper format. At the end of program ; 
execution, the driver^i^alates the- >;check4wFitiftg# facility of -the payroll 
..system-- in the following banner? It directs- the" output of the payroll program , 
to an "outout f ije. .The output consists ^of ar record, of data for each company 
employeei, >Eactt record contains,. the following "information: 

o Employee name - 4 : 

o Employee social secuiity dumber - •..*;•'' . 

* Check' date V , \vt- ■ * 

f v o Total employee earnings less .deductions - ' • ■ . • 

■ /• - *---yr' . £ ;.. . ■ ; •' />. \ * ., . 

The test driver then dumps* this information from' 'the -output -file onto a 
hardcopy; device, so that .the output can • be analyzed and v verified for " 
correctness.:...- ■ : y _ . / 

4.30.7. Effectiveness.. The iase x>f test beds' ! has proven to /be' a'mghlyV 
effective and widely V used /technique to test the operation of -software. The 
use ofLtest drivers, in 'particular, is one of the most widely usecF testing. 
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techniques* 



__4.30;8. ApBlicabili tyi; vPi is meth o d is generally ka p plicahle. .* frcm single 
nodule to large system, testing- and for .all types of ccmp^uting. applications. 

4.30.9. Learsing. In order-to build, an effective, test bed, - it is necessary 
to develop a solid understanding of the software and its; dynamic .operation in 
a system. 'ibis understanding should aid in determining what parts of the test 
bed deserve -the most -attention, fdurlng its construction In . addition, 
knowledge of the ^brnamic nature of a program in a. system is required in 
• gathering- useful intermediate outputs during program execution and in properly 
" examining, these results. j\: ■ • / , . : ■'• ." .," V- / -•• .' .- "C 

, 4.30.10. Cost. The ..amount of realism desired' in a' test bed will be "the 
largest - « factor- affecting cost. Building, a realistic test.bed may require the 
purchasing of new hardware "and the development of additional software in order 
Xx> properly simulate an entire, system, ^addition, these added resources may. 
be so specialized that they may seldom, if* 1 ever, • be used • again in . other 

. applications. ^ In ,,ftds ,way. t . very sophisticated test beds may not prove' to be 
highly cost-effective. ' ■■[ ' ^ - • x ^ V 

4.30.11.! References;? ".".;:.>•; ' %V-i ••• • 

* CI) HARJWIC^,/ R.D. ^ "The Advanced* Targeting 'studvl" SAMS(^TR-7l-W. 
Volume 1, ;«fone 1971. - •' ■ : > ; : ,:y 

"J V7"i'. * V -•• " - - .' •.. ' • ;• •. • fv 

', ; .: C2) FANZL', K'Ji, "Automatic Software Test Drivers, « IEEE Computer . 
Apra i978.. • >... .:■ - ,.;■> J ~ 
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4*31.1. Name. y«alkthroughs.^;;> ^ . .M'H' :^3 



4^31-^2. Basic features;? Walkthrojughs" (WT) constitute a structured series,; of .;- 
peer reviews -ofra system component used to enforce standards , detect errors* 
and improve development yisibnity and system gualily . They may be conducted 
during any of the I-ifecycle phases and may>aiso be applied to documentation. 
Jfe identifying featir^ of a- WT is that it is generally presented by "the*- 
creator or producer of -the material being reviewed- ratoer than an independent 
or third party. . In^addition^. because of ; the presenter's advance preparation '* 
and his familiarity with the material , r less preparation by other members is 
required. ■; •-; :y--?\iyrX r :. >v v' >.•-'• ' : ' V.-' \ v- • •„..•' 

4.31.3*1 * Information, input.- . ' ■ ^ 

•'; " at Walkthrough: Package* Ibis.. s"et ..-of 'materials ' includes all necessary - 
backup documentation for ihe WT. .Examples of. materials .made available include * 
(but are not limited "to) module flow charts, "system flow charts, tfiCPO charts 
(or other high-level t ^presentation schemes), and module listings^ tAjbrnttft 
important materials imay 'include sections • or the -I\mctional^pTOif4cation, 
System/Subsystem - Specification and Database Specification ; Xas applicable) • 
which pertain to^ the -component under ' review. Often,:, copies' of applicable 
'standards. are^ also. part of' the WT input. . '•• •;• - v ' ■ 

; / b. Questions List.-/ Some organizations which practice "a more - formal 
version of a WT . require -reviewers to submit the.compofaent to the presenter^" 
prior to the WT. This enables the presenter to be better prepared to .respond 
to the questions atth'e.WT. fi^i':, ->* $ ■.: ' k" 



4.131.4. Information- output. • ' . «..£>" S ... 

:. . . . ■'" • . .* ' - " ' • ' . ■■ • «- . .,<•. >.>» . . ' .-,-*•>>"• ♦>?••''> 

r ^ ;;. a. Action List. During the WT, a list of problems and questions 'is - 
recorded*. This action list is distributed to all participaiits and is used by 
;^e^pr^u^^ : .( changes to the component. 

; b. Walkthroughs Form. - . During the course of the WT, this form is 
completed by an ; : individual " .with v recording responsibilities. 4 The form 
identifies participants anxl'ttheir 'fia^S^b^itl^i1^\:ag^k for the WT, the ; 
decision of the ;WT Caccept "-as^is," r^ another WT), and 

assigned by all: parttioipahtis -at fhe;end ; of. ttte'WT.: .- . ,' • 

-.a, - ^Roles and Responsibilities. ' The group of individuals 
.participating in a WT afe usually referred to as reviewers. The leader of the 
WT is called the coordinator. The coordinator is. responsible for WT planning* 
;prganizatipn, : and ,jd^1^butiohA of materials^ i The WT- is: called to. order, , 
^moderated, and sunnarized by ; 1he cobrdijjatpr^. - 

■^e^^ucer (or sreviewee) is that individual who^'. i moduie^or component is. to 
be reviewed din-ing the WT. In most' cases,^ the pfoduder;, is .generally 
.responsible, for selecting the coordinator and review team (in most' situatibns; 
.sometimes .management may perfora this furc providing the WT package 
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materials tQ the coordinator, During the WT the producer initially provides a 
f^fSii ^ <te ? cription ' the moduli then leads the ^reviewers through a 
detailed, step^by-step description of the nialiae. the WT the producer 

^rahoulcp objectively consider every item on Uie^actionntst and make changes to 
yhis^producfc as he deems appropriate* .,. : " ■ ' 

1 Vpie^reyie^rs are composed of. individuals from varying backgrounds and fulfill 
; .re^nsibiTitf es ; based upon -their area of specialization. Some roles which 
^ .are fulfilled are those ixf . recorder >and -representatives of the user, standards 
- and; n^tenance ; groups* Inderal, these participants are resp^sible' for 
being fam&iar- with- the-.material being presented, submitting comments prior to 
y tbe^review^ anrastgning and contributing d WT. At the end of the 

review each must cast a vote indicating v whether the " module is acceptable 
needs revision, or is rejected., . ^ " ; • v — - . I , ' 
.1'. :-.-y. ; •;' -'..•* . . ... . > » • •*.-•'• .ty • . N < 

* .Because" of ;$he organization ' which each pisk representing, seme specific 7 
responsibilities are associated .with each, reviewer. In addition 
contributing to the WT* the recorder must ' make written note of , 
participants assembled and the action' items wtech '.result ,f rem the "review. 

' The user representative ii; often involved durJ^- early .WT»s\jof a jnoduje (i.e. ^ 
.during^ requirements^ analysis ^and design). His responsibility is to ensure 

* that the, proposed .solution is usable 'and does, in fact, :meet ; the heeds of his 
. organization.., ,. - • - • •.. .! .. \.-gy> ■■' \y -. ; v, 

The standard? representative, referred to by seme' sources as' a standards 
- : bearer,* is - responsible forchecking^/fehat the -product- being reviewed * adheres 
to^organization^standards. In some cases, he may be asked to provide input to 
^request/to deviate from a standard. - V * 

• The K n^ritenance? representative^^ v ref er^ed to- by .'some ^.sources, as the 
•Ja&tenance oracle,," must view the product f rem 'the standpoint of the groups 
who will be required to maintain the product." Items which may be of prime 

^concern to this individual are documentation and program comments, program 
v:WS u » ial 4ty modularity, naming^ebnyentions^ and data decomposition. ' 

* 4f b * ^ e EProcess.; Many •- organizations practice walk-throughs which 
.differ radically in formality, y The process descrlbetT^Ln the following 
.paragons- falls at the midpoint between these -extremes^ There are four basic 
steps in the process. ' . ,? :.r' •' *.** ;. ..■--^T. ~'l . 

.. . Scheduling. When the work item module is very near' completion 
(including documentation) , the producer notifies management and selects 
; the WT participants. The WT date is agreed upon and facilities'- are. 
scheduled. The WT should not exceed a^Tours and is best is kepl^tblless 
■> than. 1 hour. This Implies that fc work Item Jcs of- manageable size.* " > 
% - ,* Sources ^suggest r the following guidelines for-Wbrk package* size:- : v-V - 
o'S-W pages of specifications for a requirements WT, ' v ■ 
P 1-5 structure charts (or HIPO diagrams; for a preliminary or " 
T - detailed design WT; :: ' ' ' - • 

i; • • ip. 50-100 lines of Code for a code or test WT. 



■ > ••^*^^J l ^P ai ^ t i9 n ^ • ih^ prpd^rlco appropriate .infonnation for us# ? 
• : n .at=tte^iff^r ^ for distribution. Each : v 

- ; ; ir^yiewer. stedTes the matermsT^Effi^a "note ^^sliSJor^eom&te^ 
/ - «yrces estimate that a maximum, of >1 hou r preparation by f ggWers 

. ; \' .is'' necessary #v • •- • ~ : ■ ■ .•' • . • • : '-.<Z/'-- 

> . ;; . h . ' * r< vv^ ''i/.''* ; : -•.*-'.".,.* -v: '■ 

,;<. <KMucer uses test date to^ simulate the operation of tee ccmponerit. ^ 

• Each jpec^c^on;; design -phrase, or ;line of, code is reviewed, * The 

; : . recorder documents commente or questions usng tee action list: ' Each ; 
V •• reviewer "signs the Walkthrough, documenting the decision of the - < • ' 

■ • -2««ting (accept, product, asris r accept with modification.' -or 'reject) V . 

-r The recorder .provides a copy of action- list to all participants v and • • 
v ; . supplies a copy, of the Walkthrough Form te management. * \>c. 

• " 4. Re-Work." The producer /reviews each action item, making product - 

- cban 8 es ^ he feels necessary. He may decide to implement all/- part V - 
; . - or none- of the suggested, changes. No follow-up is held to ensure that-" 
■■ suggestions are incorporated; it is assumed- that the producer is in th&v* 
best position, te;make implementation 7 decisions. 1 Major items on the*. ' ^ ' 
rv action list jmay^; summarized at the next JET for the module. . *; v 

Jc 3 I;«* Example. One week prior -to. completion, 'of <x>ding .of a module 1 of " 
lines, the producer notifies his line, manager, of the need for »a WT : ; 
Upon manag«fcent approval the producer selects a cooBinator. (one of the lead 
*^ a h??^. rr0fa ' development shop), a standa^s representetive Cfrom tee 
Quality Assurance group) , a maintenance representative (from the Produst^n 



Program^ orgajaization) r and a user represortative (from the group requesting 
i*e syst^y.^ Three days prior to the inspection he notifies the - : ^bSitfaatoS 
of. tee- .planned WT and suggested participants. At this time he gives the 
"22^S S- eS u °t m Program listing '.(including ? comments) , : ' a 
systetns^leyel flowchart depictihg how, it -interfaces, with other modules, a date-; 
^ionary,^ yt; o^ tes^date items, and a section from the -Functional ' 
speci^cation detailing tbe user requirement associated with the -module. - 

- The coordinator notifies tee selected participants, receives their commitment" 
to attend and distributes to^each a„copy of the materials furnished by the 
producer. . , ; -' r; - • i u * ^ . 

T ^ch participant "reviews ^the materials, i The-ste^iards representative f^rids 
two instances^ of , deviations from published standards and notifies -the- - ; 

^ n ^ user representative 

-verifies; that the,; code-, addresses1<; eacfr^designed aspect- by Reviewing tee 
^proceeo^s of tee previws;;d^ each: requirement 

r 33 . been addressed and. notifiesl tee coordinator thatr he finds no errors arid 
feels that his presence"- is not required for . the code wiLkthrajghV The 
naintenancA repr^tetiye 

a note to inquire about the structure c>f tee" date files> 



,-- . 



The. WT begins wite ,a ,brief ihteodudtion by the coordinator, who teen turns the 
,.review over to tee produter.- Be'uses the system flowchart to^give-a summary 
of^tee^neticm^ line-^-^e ^through- the 
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code, aslng the selected test data. .;. Upon reaching the lines of ; concern to the 
standards, representative,; a; brief discussion occurs to explain 4he reasons for 
8&- : S?^ tt .?-f, : ^^*tlii.s: instance, the reviewers are satisfied 
IS ^^SS ^^T 1 " ^^ ■ r yo^-^otesiK?n--tfae--acti^ Tlist 
^^L^^ 18 pro ^ ee< ^: The maintenance representative points out one line 
of highly .complex code and suggests that it be broken up into two less complex 

?^\,^? l -? nt cannot ^ inmediately reached, so -the suggestion Is added to 
tne action list. .•>..•« > * ^ ; ' * >•>-., 

At the end of vthe-module review T the : coordinator "seeks* a? decision from the 
reviewers about the module. They agree to give their- approval, providing that 
the suggested changes are made and that the producer will further investigate 
??,i£ f|3ec J of . ; . breaking up the 'complex line of code. Each signs^he " 
. Walkthrough form and -the meeting-is adjournetL ; ^ • - ^ - . ,^ 



The recorder distributes a. :copy 'of : the. action to all participants.' The 
producer makes the changes he feels are necessary. He runs -a benchmark of the' 
module with t^ e complex code and again with the Code broken- down. -Since no 
; significant- loss of efficiency resulted, he modifies, the .'code. The module ^is 
now ready for unit test which may be followed by another WT. . . ^ 

4.3U7. .Effectiveness. Studies have been conducted ' which identify" the 
following qualitative benefits of Walkthroughs; -v" • " 

o J higher status visibility" ; / : \ ■ ■ ■ ' ' •' .' . ' ;' • ' . • 

a o decreased debugging* time ... " . ' t 

* 0 early deletion: of design and analysis errors which would be^ much more- 
costly to correct-in . later develoment phases " : : " 
o identification of design or ^;ode inefficiencies ; '• 
6 r ensuring adherence ; "to standards 
«.->•. o increased* program readability - 
fif ,/ 0 ^creased user satisfaction - 

*o communication, of new ideas or 'technology t • '■: ' * ' 

.. o increased maintainability' " 



Little -data- is available which identifies the quantitative benefits 
attributable to . the use "of Walkthroughs. -However, '^hev source estimates that ; 
the number of errors in production programs was, reduced by a factor of- tent 

• V ... .- " •■>--- •'- .:•' "*•••'■•;."•• ,, \A ; . 

4.3J.8, Applicability.^ The Walkthrough, isv applicable -to^ llrle"- or ' small> ~- 
projects^ during all development biases; and Is not limited by project type" or*- ■ 
complexity. •. ■• , •**"»•. *;\ , " • . \, 
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'$.31.?. Learning. ThevWalkthrough J does not require special training to 
implement. However., ' experience has shown that the -effectiveness of the 
Walkthrough iricrea^ increases.- - 

^ItfQ&tCfs^j.&.p^m requires no special tocds or equipment .to 'implement. 
The direct costo are equal to" "the. expensed resources 
involved.., ...» . • . T -• • 
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■"<;.'• . V , ; 7 : ( ty ^eode^Beadlng; Structured Walkthroughs and -Inspections*, IBM IPTO ! 
Support Group^tforld?^ 

• ; ^March^^i^^-;>; : /••f;- ^fy .'^'vr.v . ;\ y / ; '.. 

: _ t2> ■ FAGAN, M. E., "D^iglr and Code Inspections to fifiuce ErroT^ in 
; Pjrog^m jfeel v opment», IBlt •Systana- journal, No. 3, , ffl6. ; 

f.'v'it- -(3);' FREEWttl^ IK.; P; f;ahd,ttffiWBM^, .M.V PEthno - -Technical Review 
Handbook", Ethnotech, Inc. , 1977. * ; V ■> " " •. r, 

* ; (4) ^DALY, E v B£, -Management of - Software i: Development", IEEE 

Transactions on ftaftuare ^nPAi^nff^ May IQ77.: ^ , V / ' 

• (5)33»fflfcBH», Ben, \"&ftware Psychology..- Human Factors" in Computer * 
and Information Sysie^," Winthrop Publlshl^ - 
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GLOSSARY 



BLACK BOX TESTING : \ see- FUNCTIONAL TESTING 



B^NDARY VALUE ANALYSIS: a Section technique in ;which test date is chosen 

^l5f ; ---^2^ ? i- 1 '^5P dl,, ? J ^^; : -^'? : -??* t ^~ of input ^omain' Cor output range) 
classes, data structures, procedure parameters* retc. Choices^ often ^lude : 

'v 1Ba ?™»,. "on™^ j and trival values <r^paramet^ 

; ^called stress testing. : - . •*•>•:*;,£.:• . .sbjj. £ 

; v BRANCH TESTING: a 'tes^ Method satisfying' coverage criteria thatfcuire- Tor " 
^i 63011 ^ 60181011 P? int » Possible branch to; executed at least ogee. ' v 

r CAUSE-EFFECT GRAPHING: test date selection technique. The' inputs and: outputs 

; . set of inputs is chosen avoiding the testing-^ multiple inputs, which cause • 
identical cutout. . ' -\" ; : -: : • f>: .• ■ >, , ' • 



COMPLETENESS: . .the property ,toat all necessa^ parts .of the entity in question 7 
-jre included.^ . Completeness^ of afproduct is often used to express the fact 
/ r^St aLl requirements have been met by the product. " 

CONSISTENCY:, the property of logical coherency amoung , constitutant' parts. 
; Consistency jnay also lte expressedr as adherence to a given set of rules;/ 

; v : CORRECTNESS: the extent to which software is free from design and coding 
•*> oefecto^ i^^ult free. It is also the extent to which Software meets its 
■ specified, requirements and .user ^objectives. ' ; (IEEE Software Engineering' 
.; Terminology;'..^' V- ... .^4- " • •. • 0 

l. ..... -.. ■ . 'r,f>V .{-.v.: • . .'J ;,-..-">•>.... . . » . . . • •. . ■.. ... -• «••.? 

-DEB^^ logical errors detected 

. •during^ ceding. Wito the primary go^ of code, . 

digging; -shares wito testing certein^tecfiniques and strategies, but differs s in - 
-its usual:; ad^hcc application and local scope. " r *• :■■ --* . s. 

f^Monalr : analysis^, (see v FUNCTIONAL TESTING) - extended to include desira 
- luncticiis as well as requirement functions. . 

DRIVER^-, ^e which sets up an environment and calls a module for test. 



. . tC" ANALYSIS: involves execution or simulajbion of a \ development phase" 
- prodMct. It detects errors by .analyzing the response of a product to* sets of" 
^ input date. ' ' .'••,<•' ' - i.>. 

/" EXTREMAL TEST DATA: .test date that is at the extremes, or boundaries^ of : the 
j : domain^ of an input -variable or 1 which produces'; results at toe touhdaries of an 
v^outout domain^ • . . - • - • v -V-'-*.';'" / : 



. *FDRMAL*- ANALYSIS: uses rigorous mathematical techniques to analyze "the 
algorithms 4 of a, solution^ The algorithms may be analyzed for numerical 
properties,' efficiency, and^pr "correctness. . 
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FUNCTIONAL TESTING.: application of test data > derived from the specified 
fXinction^ the final program structure. ' ■ ;g 

INSPECTION^ ^ a manual analysis technique . in which the program (requiranents T 
design, or code) is .examined : : in a very formal and disciplined manner to 

; discover-~eriws..S/- >' : ; : ■ ■ 

INSTRUMENTATION:' the insertion of additional code' into the program in order 
to; collect information about program behavior durihg program execution. . . 



mGLm INPUT (TEST DATA FOR INVAMD / INPUT DOMAIN) : . test data that -2±es" 
outside'- the domain of the program's function. . \: ,. ' ?• 

: " PATH TESTING : a test method, ^satisfying coverage criteria that each logical 
••• path .- through the program "be tested. . Often paths through the program are 
grouped into a finite set of classes; one path frcm' each class -is then 
tested. ■: / ' r Jz :.. ', v : /'"•;/..'• ;.. ■' - 

^ROOF OF CORRECTNESS: .the use of techniques of mathematical. ' logic to infer 
)hat a relation between, program* variables assumed true at program entry 
Implies that another relation between program variables holds at program exit. 

REGRESSION TESTING : testing of a previously validated ^program which has been 
modified for extension or correction. ^* 0 

SIMULATION : <use of an executable model to * represent the behavior of an 
object. - During testing the computational hardware, the 'external environment, 
. and even code segments may be simulated. ^ 

SPECIAL":TEST DATA: test data based\on input values that are likely to require 
special handling by the program. ,N-^._ - » -:• • • - 

CTATEMENT TESTING : a test -method satisfying the criterion that each statement, 
in a program be executed at least, once during program testing. 

STATIC ANALYSIS: -direct analysis of the form and structure of a product 
without executing the product. It may be applied to the requirements, design 

or "code.' • ,;. • ' • . . 



STRESS TESTING: see BOUNDARY VALUfi ANALYSIS. * , ^ * < 

^HJBz special code segments that when invdced by a code segment under ' test 
will siimaate the behavior; of /designed and specified modules nt>t yet 
. constructed. ; • v ■ ...v, . • 

SYMBOLIC EXECUTION: an analysis technique ^ a Symbolic expression 

. for each program, path. v : : ^ v \ 

TEST DATA'SET. : set'of input kemehts used in the testing process. 
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TEST DRIVER: , ; .a prwam whicfr (tfrects the ex&ution ^M'other-progran against 
a collection : of test data sets; Usually, the > test > driver records and 
organises the output generated as the tests are. run. :< . . 

XPST HARNESS: see TEST DRIVER.'- * - v : r : 

'^^••i-: ; \ : v -Ti^^"., • •> ' . Z . At- 

testing: examination of the behavior: of a program by executing the program on 
sample data sets.; • ' •■ X\- .' ■.■<■-, ' ■ .yV vy 

'f----' ; '--- v ^'-'v^ ; : '•.••••'•••':'?.v'-- > ' : --.' \ - 7-V; ■■ c::.^ 

VALID INPUT (TEST DATA FOR A VALID INPUT DOMAIN) : test data that Hies within 
the domain of the function represented by the program. .* 

.VALIDATION: determination N of the correctness of the final program- or software 
produced from a;, development^" project with respect to the user -needs and 
requirements. ' * ' 

■ " . • :.• . v '•' • ; \ . ■ :& ? : . ' y. ■ \ ,; ■ 

VERIFICATION: in general,; the demonstration of consistency, completeness, -and 
correctness of the software at each stage -and between each" stage of the 
development lifecycle. .••*.." , . '. "." -vVy • 

WALKTHROUGH :—a jnanual analysis technique In which -the module author describes 
the module's structure and logic to an audienee of colleagues. . . 



NOTE: Most of the definitions above are'fran:" •• • ' 

ADRION,W»R..,BRANSTAD, M. A. ,and CHERNIAVSKY, J. C. , "Validation," Verification, 
and Testing", NBS Special. Publicaiton 500-75. . / 
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